investigating....

Cool, thanks duglin. I moved to its own issue as it seemed more specific to --env-file than #11443

yup plus now we won't forget about it ;-)

In looking at this I think converting \n into a LF for the env-file might be an issue.

In all of the examples we're playing with I believe its bash that's actually converting the \n into a LF and not Docker or even normal string analysis. For example, $'hello\nworld' doesn't work in sh. And it only works in the '-e' case because we're letting bash do it for us before Docker sees it. I don't think its possible to get a LF in an env var from within a Dockerfile either.

I wouldn't mind making the change if we could justify it as a normal sh thing to do, and if it doesn't break people, but I'm not sure we can.

ping @erikh @tiborvass what do you guys think?

With Compose, for example, you get this behavior:

web:
 build: .
 environment: 
   TEST: "hello\nworld"

docker-compose run web ruby -e "puts ENV['TEST']"
hello
world

and this is because in YAML, if you have a quoted string with \n in there, they will be interpreted as newlines and not literally:

>>> import yaml
>>> filename = 'docker-compose.yml'
>>> with open(filename, 'r') as fh:
...   yml = yaml.safe_load(fh)
...
>>> yml['web']['environment']['TEST']
'hello\nworld'
>>> yml['web']['environment']['TEST'][5]
'\n'

My ugly hack at the moment to workaround this is to use echo -e to evaluate the line before passing it to docker https://gist.github.com/hudon/149466af21dfc52fdc70

that's really interesting and consistent with the pattern of "something else before Docker converting it to a LF".

I'm really torn. I do want to help and see if we can find a solution for you but at the same time I don't want to muck things up either by adding complexity or break existing users.

Just brainstorming.... one option I thought of would be to add some kind of comment/directive to the env-file to indicate that you want escaped sequences converted. This shouldn't break backwards compatibility. One concern I have with this though, is that we now have this ability via env-file but not via -e or even within Dockerfiles. Granted we could claim that if you want support for it via -e then use bash and $'...'. As for Dockerfiles, I guess we could add support for $'..'. Dunno.

Would like to hear from @tiborvass, @tianon and @erikh on this entire thread though.

Another alternative is the ability to declare a variable on multiple lines, which'd allow you to put an actual newline in there. If " is encountered, slurp value until next ". So you can do:

# .env
TEST="hello
world"

tricky. That certainly would make it align more with normal sh parsing.

Of course, it would mean another parser since I don't think any of our existing ones support crossing lines w/o a \ at the end. But hey, what's one more parser! :-)

@tianon @tiborvass @erikh any comments?

Sorry for the inconvenience but I would rather go the conservative route on this one :(

The problem can be solved with echo -e like you mentioned and I think the benefit of keeping env-file simple is higher.

I'm going to close this issue, but feel free to continue the discussion and speak up if you disagree.

I think --env-file shouldn't just take the file literally. Probably a bigger issue is how it also reads quotes literally:

#.env
TEST="hello world"

$ docker run --env-file .env container  ruby -e 'puts ENV["TEST"]'
"hello world" #actual
hello world #expected

w/o stating whether I think we should be smarter or not while processing these files :-) since we just blindly take the string after the = as the value, there's no need for quotes at all to escape things.

@hudon yepp, reading quotes literally threw me off.

Another alternative is the ability to declare a variable on multiple lines, which'd allow you to put an actual newline in there. If " is encountered, slurp value until next ". So you can do:

That would be great from a UX perspective too. Right now, trying to pass something nested, e.g. json, is quite a pain to do on one line (if you want any readability).

# doesn't work
SETTINGS={
  "test":"test",
  "someOtherLongKey": {
    "ohIsThisAChildOrNot": "IReallyDontKnow"
  }
}

# works
SETTINGS={"test":"test",  "someOtherLongKey": {"ohIsThisAChildOrNot": "IReallyDontKnow"}}

Given that I can source a file with the json defined within single quotes made me think it would be possible to declare variables on multilines with --env-file.

export SETTINGS='{
  "test":"test"
}'

If nothing else, the error message for the first example I gave "docker: poorly formatted environment: variable '"test":"test"' is not a valid environment variable." could be more helpful if it contained a description of what's allowed.

I stumbled over this issue today. My use case is trying to load a pair of PEM encoded RSA keys without exposing the keys via the process table with the -e flag. This led me to --env-file which I pointed at a file I also use both with bash source foo.sh and systemd's EnvironmentFile=foo.env directive.

It would be great if all three of these use cases could work without having to modify the file to make it compatible with the different parsing behaviors of the three. I suggest a simple double quoting like systemd does, with preservation of whitespace between the double quotes.

As an example, I'd like this file to work with docker --env-file:

RSA_PUBLIC_KEY="-----BEGIN RSA PUBLIC KEY-----
...
-----END RSA PUBLIC KEY-----"

RSA_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----"

I think it's fair to expect --env-file to handle quotes around values e.g. FOO="bar", and from there support for newlines is obvious / automatic. It'd be nice to match other widely used env-file tooling e.g. https://github.com/bkeepers/dotenv/blob/master/lib/dotenv/parser.rb and https://github.com/joho/godotenv/blob/master/godotenv.go

A pretty compelling use case for .env files is storing encryption keys outside of a code repository. Not being able to include new lines in values means being unable to store PEM encoded RSA keys without implementing some hackish workaround as described above. Moreover, this is inconsistent with the "environment:" section of a Docker compose YML file, where "\n" is actually replaced by a new line. I personally don't think that having a slighly better parser for env files is unnecessary complexity.

If you are worried about backwards compatibility, a possible solution would be to add a new ".env2" file format which supports quotes and new lines.

1 we noticed an issue where docker --env-file's interpretation of our /etc/environment file differed from bash's, where we were using . /etc/environment to evaluate the file.

1 for --env-file handling quoted values; I need to handle some secrets containing newlines.

1 same here. We store private keys as env variables and this would greatly help us

1 same here. We store private keys as env variables and this would greatly help us too.

1 for --env-file handling quoted values; I need to handle some secrets containing newlines.

It should be possible to use @duglin's quoter in the builder, maybe.

On 20 Jan 2016, at 23:59, Marko Stipanov wrote:

1 for --env-file handling quoted values; I need to handle some
secrets containing newlines.

---

Reply to this email directly or view it on GitHub:
#12997 (comment)

It would be extremely helpful and common-sense to have Docker parse these files just like the shell would as depending on the scenario one will sometimes source the file, sometimes use --env-file

1

1

Same use case as others: encryption keys being loaded, would prefer to be able to use --env-file for that.

1 Same usecase here

1

Spaces are problematic too.....

docker: poorly formatted environment: variable 'KAFKA_HEAP_OPTS:'-Xms128M\ -Xmx256M'' has white spaces.

@tiborvass Please follow up to the conversation. There have been 20 messages since you have closed the issue.

@grizmin your error seems to indicate you're escaping values. The env-file format simply takes everything after = as a value so having this in your env-file should work (no backslashes etc):

KAFKA_HEAP_OPTS=-Xms128M -Xmx256M

@gajus I suggest to open a proposal that adds this feature in a backward-compatible way, see for example #12997 (comment), or perhaps an approach that allows options to be passed. Such a proposal can be discussed (but no guarantees)

However we should really guard against feature creep, and not try to replicate things that are already possible (e.g. options that are already possible by combining some shell commands). There's a real concern for adding a lot of complexity, and before we know it, we're re-inventing Bash

1

@tiborvass: I'm going to close this issue, but feel free to continue the discussion and speak up if you disagree.

What's the point in continuing this "discussion" and speaking up, if we haven't had an official response since, with quite a lot of requests for this feature in 1,5 year time?

@JeanMertz #12997 (comment), #12997 (comment). Changing the format for the current --env-file flag will most likely lead to a backward-incompatible change. And there's no common syntax for env files (#12997 (comment)), so the most likely option would be to add another flag.

Note that if you have an environment variable set locally, you can already do this;

export FOOBAR=hello
docker run -e FOOBAR busybox sh -c "echo \$FOOBAR"
hello

(omitting = takes the value from the current environment)

Quick and dirty solution and possible temporal workaround is to replace string '\n' with actual newline symbol directly in code
here is python:
'private_key': os.environ['GA_PRIVATE_KEY'].replace('\\n', '\n'),

mesos 1.2.0 creates container via env_file

marathon-acme for letsencrypte sert and marathon-lb for balancing

i patched marathon-lb to replace \n in config to newline

set newline as '\n' in mararthon LABELS for marathon-acme app
and get '\n' in haproxy.cfg on marathon-lb

https://github.com/LuckySB/marathon-lb/tree/workaround_bug

1

Here is another way of passing env vars to the container which is newline-safe:

ENVS=`env | sed -n '/^[^\t]/s/=.*//p' | sed '/^$/d' | sed 's/^/-e /g' | tr '\n' ' '`
docker run $ENVS ...

$ENVS will look like: -e VAR1 -e VAR2 -e VAR3 -e VAR4

1 (this is really basic, and annoying.. this should be fixed)

Same here. Please enable newline support.

Wait... How did sharelatex get to run their example env-file while it is not supported?

@louwers you linked to a Docker Compose file, not an env file as supported by the --env-file option

@hudon Yes thanks, I figured it out by now... 😅

👎
such basic problem could not be solved for so long time!
really disappointed!

1

1, please implement it

1

I am locking the conversation on this one, because complaining and " 1"'s are not constructive, and will not get this change implemented. This repository is open source, and contributions are welcome; if this feature is important for you, invest time to get it moving. From my earlier comment #12997 (comment)

I suggest to open a proposal that adds this feature in a backward-compatible way, see for example #12997 (comment), or perhaps an approach that allows options to be passed. Such a proposal can be discussed (but no guarantees)

However we should really guard against feature creep, and not try to replicate things that are already possible (e.g. options that are already possible by combining some shell commands). There's a real concern for adding a lot of complexity, and before we know it, we're re-inventing Bash

When writing such a proposal, be sure to describe what use cases it is resolving, and if other options are already possible; for example, docker services now support secrets (docker secret create), and configs (docker config create) (user guide: https://docs.docker.com/engine/swarm/secrets/). Templated configs are being worked on in #33702.