#1571

Overall I'm in agreement. Some questions and comments

• Why do we need to provide a new binary for this function as opposed to making the config plugin as it is today more robust?
• I'm not clear on the difference in interaction regarding how a plugin would interface with config vs. an app.
• 👍 on config VCS
• Involving encryption include key management. I think this is best left up to the reader for now
• I think it's fine to move nginx config into this space as well
• Why serialize to anything? I think this makes the interaction more complex unless we build that into the aforementioned binary. Doing it in bash seems hairy and overtly complex. We need a solid use case for taking this on.
• I'm not sure it's up to us to save a user from a "shitty" plugin/app/user.
• Again, why create a new binary dependency?

I guess moving it to a separate binary would allow us to iterate on that quicker than we would on the config plugin. As well, handling input/output in bash is kind of annoying, and it would be nice to be able to use a higher-level language to handle this (not to mention it could also decrease the amount of logging output if we write it correctly).

Interesting ideas as I think also that we need some sort of central repository for managing config. Indeed it needs to be versioned somehow with git for example. We need to have a way to bind a revision of an application config with a tagged image to be able to do rollbacks and things like that (like Heroku) :)
Another thing I saw the other day while playing with their addons is that when an addon declares a config var on your app, you can’t manage it directly. Only the addon can manage it so this is an interesting lead to explore to avoid unwanted rewrites and thing like that.
Separate “config stores” for nginx and plugins seems a good idea. As for the serialization, if we really want that, maybe we can use toml?
For a new binary, I don’t know if it’s necessary nor if Go is a good choice ;) (I find it way too verbose among other things)

Otherwise 👍 :)

@josegonzalez so looking this over, I have a few questions:

• Do you have a copy of the source for the "binary"? I assume this is just because strings are hard to handle in bash?
• You are proposing to move all app data out of ~/dokku/, to /var/lib/dokku/config/apps/ correct?
• For data not committed in git; any changes to quasi-ephemeral data like environmental vars (ENV_FILE)?

Moving the location of the apps is simply handful of greps, is there an advantage to a new location or is it just to force adoption for thirdparty plugin authors? In the same line of light, excluding all backup abilities (manual or automated), what other advantages are gained by the dokku community with these changes? One less part written in bash is a good answer too. 😉

• I haven't written any code for said binary.
• Not the git directories, just any config files. For instance, we'd no longer be copying nginx.conf.template or whatever into /home/dokku/DERP/, and instead to the /var/lib/dokku/config/apps/DERP/ directory.
• Any files that dokku creates that are config would somehow go in this new config dir.

The advantage is separating all our config from the bare git directory, which allows us to both simplify backup/copies of applications, but also removes any doubt about what files go where. In the past we just never gave anyone any direction on where to put stuff, so shit has gone everywhere.

I started giving this a good, hard look. I believe once the config changes from #2751 land, we can look into this. I'd be down for something like an attributes or properties plugin. It could be a minimalist wrapper around something like bash-filedb, while allowing users to swap in/out the underlying binary that is called (in case they want to use something else, like redis or vault).

I'm not sure how data migration would work - maybe some sort of read-through layer that allowed us to fallback to a config key? - but thats something to think about. I definitely want to make sure we move all config out of the application repositories though.

The way forward here is the properties functions - implemented in golang and shell - which we should use to migrate plugin-specific config over to new paths. An example of how this could work is in the git install hook, which migrates DOKKU_DEPLOY_BRANCH over.

Whats the status on this? Looking for a way to backup and restore my dokku apps,plugins and configurations.

@mcohen9 Hi! Any "status" would likely be posted on this issue - either in the form of a comment, linked pr, or issue closure.

As this is an open source project, any change in "status" to an issue will have to be contributed by our users - this includes you! - or project maintainers at their leisure. If you'd like to sponsor the work on this, it's probably ~20 hours of my time to move everything over to the new system, so feel free to email me if thats something you want to consider.

As always, pull requests welcome!

Open question: Where should application repositories be stored? /var/lib/dokku/data/git might be reasonable. One thing moving the repositories breaks is how applications are listed/verified as existing, so we'd want to track that separately as well.

@josegonzalez anything I could do to help?

Biggest thing would be to see what files are being read from the DOKKU_ROOT dir that aren't git-related and move them to the appropriate place in /var/lib/dokku. I haven't gone through and documented these just yet, so that would be a good first step.

Update:

• IP/PORT files can probably be computed on the fly and removed.
• ENV file should move to /var/lib/dokku/config/config/$APP/ENV
• DOCKER_OPTIONS_* can move to /var/lib/dokku/config/docker-options/$APP/phase-*, where * is the name of the phase. This will also allow us to support process-specific options at some point.
• nginx.conf files (and certificates) should move to /etc/nginx/dokku.d (and tls somewhere appropriate). There is a ticket open for that (Move nginx config to /etc/nginx/dokku.d #3443).

I'm going to try and move the ENV/docker options files and refactor the ip/port stuff in the next PR, after which we can close this issue as everything else is tracked separately.