Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restarting app causes 502 until full stop and restart of entire docker-compose stack #1289

Open
Redsandro opened this issue Jun 12, 2019 · 4 comments
Open

Restarting app causes 502 until full stop and restart of entire docker-compose stack #1289

Redsandro opened this issue Jun 12, 2019 · 4 comments

Comments

@Redsandro
Copy link

Redsandro commented Jun 12, 2019 •
edited
Loading

Problem:

When a request from shodan.io crashes the smtp-server (see nodemailer/smtp-server#128) in my api app served over https, it is automatically restarted due to docker-compose.yml restart: always.

However, the app doesn't work after restarting: nginx-proxy returns 502.

Expected

  1. The automatic restart should have fixed this.
  2. When I do docker-compose restart api and the logs shows it's listening, I'm still getting 502.
  3. With docker-compose restart restarting everything including nginx-proxy and letsencrypt-nginx-proxy-companion, the logs show listening, but I'm still getting 502.

What works:

A full docker-compose stop && docker-compose up -d brings up the app and everything works. I don't understand the effective difference between docker-compose restart and docker-compose stop && docker-compose up -d, which puzzles me.

But I need to automate this error recovery. I thought restart:always should be sufficient. Is anyone else experiencing something similar? I searched, but I am human. Are there any known issues related to this?

docker-compose.yml

version: "3.5"

networks:
    proxy:
        external:
            name: proxy

volumes:
    conf.d:
    vhost.d:
    html:
    certs:
    db:
        external:
            name: test-db

services:
    nginx:
        image: nginx
        container_name: nginx
        restart: always
        networks:
            - proxy
        ports:
            - 80:80
            - 443:443
        volumes:
            - conf.d:/etc/nginx/conf.d
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - certs:/etc/nginx/certs
    nginx-gen:
        image: jwilder/docker-gen
        container_name: nginx-gen
        command: -notify-sighup nginx -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
        restart: always
        networks:
            - proxy
        volumes:
            - conf.d:/etc/nginx/conf.d
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - certs:/etc/nginx/certs
            - ./nginx.tmpl.go:/etc/docker-gen/templates/nginx.tmpl:ro
            - /var/run/docker.sock:/tmp/docker.sock:ro

    nginx-letsencrypt:
        image: jrtest/letsencrypt-nginx-proxy-companion
        container_name: nginx-letsencrypt
        restart: unless-stopped
        volumes:
            - conf.d:/etc/nginx/conf.d
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - certs:/etc/nginx/certs
            # Docker socket for detecting app reboot
            - /var/run/docker.sock:/var/run/docker.sock:ro
        environment:
            NGINX_DOCKER_GEN_CONTAINER: nginx-gen
            NGINX_PROXY_CONTAINER: nginx

    app:
        image: redsandro/test-app:0.4.0
        container_name: test-app
        restart: always
        depends_on:
            - nginx
            - nginx-gen
            - nginx-letsencrypt
        networks:
            - proxy
        environment:
            VIRTUAL_HOST: test-app.example.com
            VIRTUAL_PORT: 8080
            LETSENCRYPT_HOST: test-app.example.com
            LETSENCRYPT_EMAIL: (...)

    api:
        image: redsandro/test-api:0.4.1
        container_name: test-api
        restart: always
        depends_on:
            - db
        networks:
            - proxy
        ports:
            - '25:25'
            - '465:465'
            - '587:587'
        volumes:
            - certs:/certs
        env_file:
            - api.env
        environment:
            TLS_KEY: /certs/test.example.com.key
            TLS_CERT: /certs/test.example.com.crt
            TLS_DHPARAM: /certs/test.example.com.dhparam.pem
            VIRTUAL_HOST: test.example.com
            VIRTUAL_PORT: 9080
            LETSENCRYPT_HOST: test.example.com
            LETSENCRYPT_EMAIL: (...)

    db:
        image: mongo:4.0
        container_name: test-db
        command: 'mongod --smallfiles'
        networks:
            proxy:
                aliases:
                    - mongodb
        volumes:
            - db:/data/db
            - ./initdb.d:/docker-entrypoint-initdb.d
        environment:
            MONGO_INITDB_ROOT_USERNAME: jack
            MONGO_INITDB_ROOT_PASSWORD: ********
            MONGO_INITDB_DATABASE: test-db
        restart: always
@DekiChan
Copy link

@Redsandro did you find a solution to this?
I found that the problem is that once the app container gets restarted its IP is changed, but proxy doesn't know about it and just keeps sending requests to old IP of app, though I haven't found a solution yet.

@Redsandro
Copy link
Author

@DekiChan I did not find a solution yet, and I still have this problem.

I did wrap as many try-and-catches at every level so the container never ever ever ever crashes. But it still happens now and then. Some upstream vectors just can't be anticipated 100% of the time.

@DekiChan
Copy link

DekiChan commented Sep 3, 2019

@Redsandro, quite embarrassingly I figured out we're using nginx:alpine and not jwilder/nginx-proxy (we usually do use it on other projects).

However, I solved the issue by using combination of resolver directive (default docker DNS resolver at 127.0.0.11) and using variables in proxy_pass, which forces Nginx to re-resolve hosts. For more details see this Stackoverflow question and this Serverfault answer.

@zibellon
Copy link

@DekiChan

YOU ARE THE BEST!!!

Nginx INSIDE Docker
metrica-agency-front-dev - Container with my web-site. Inside PORT = 3000
Full: http://metrica-agency-front-dev:3000

My solution is:
resolver 127.0.0.11; <— define resilver
set $containerAddress "http://CONTAINER_NAME:PORT_INSIDE_CONTAINER"; <— define and set variable

location / {
proxy_pass $containerAddress; <— Use vaiable !!!
...other settings
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Redsandro @DekiChan @zibellon