Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Also Add Password Changing Guidelines #22605

Open
InfoSecMastermind opened this issue Apr 9, 2024 · 26 comments
Open

Also Add Password Changing Guidelines #22605

InfoSecMastermind opened this issue Apr 9, 2024 · 26 comments
Labels
documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers self-hosted Issues related to self hosting

Comments

@InfoSecMastermind
Copy link

InfoSecMastermind commented Apr 9, 2024

Improve documentation

Link

https://supabase.com/docs/guides/self-hosting/docker

Describe the problem

When password is changed like for postgres in .env , It causes an error when started again docker compose up -d , there is no mention of fixing that or any passwd changing guidelines.

Describe the improvement

After many hours i found out you need to docker compose down -v and also rm -rf volumes/db/data/ sometimes. Then docker compose up -d to change it without error.

@InfoSecMastermind InfoSecMastermind added the documentation Improvements or additions to documentation label Apr 9, 2024
@encima
Copy link
Member

encima commented Apr 10, 2024

Thanks for opening and sorry for the hours you spent!

It is probably good to add this for most configs here as changing any of these will often require a restart. PRs welcome!

@encima encima added help wanted Extra attention is needed good first issue Good for newcomers labels Apr 10, 2024
@InfoSecMastermind
Copy link
Author

As I also commented on the latest commit for this guide.

After updating the POSTGRES_PASSWORD in .env , it results in an error with docker compose down and then docker compose up -d instructions. The changes only took effect for me by first docker compose down -v then rm -rf volumes/db/data/ and then docker compose up -d. The screenshot for log (docker logs supabase-analytics) of the error is attached.

log

I think its more of an issue, maybe someone else should also test it.

@IcedTea2K
Copy link
Contributor

IcedTea2K commented Apr 13, 2024

It is probably good to add this for most configs here as changing any of these will often require a restart. PRs welcome!

On that note though, maybe we should recommend storing POSTGRES_PASSWORD and sensitive data with docket secrets instead of using .env for better security. Check the tip note here

@IcedTea2K
Copy link
Contributor

Looking through the docs again, there's already a section on restarting services after making changes to the configs (https://supabase.com/docs/guides/self-hosting/docker#restarting-all-services). No changes maybe required

@InfoSecMastermind
Copy link
Author

Its not working for me as per those instructions in the docs, especially when dealing with changing POSTGRES_PASSWORD in .env.

As I also commented on the latest commit for this guide.

After updating the POSTGRES_PASSWORD in .env , it results in an error with docker compose down and then docker compose up -d instructions. The changes only took effect for me by first docker compose down -v then rm -rf volumes/db/data/ and then docker compose up -d. The screenshot for log (docker logs supabase-analytics) of the error is attached.
log

I think its more of an issue, maybe someone else should also test it.

@IcedTea2K
Copy link
Contributor

The link to your log might be broken or private. Do you mind taking screenshot or sending another link?

@InfoSecMastermind
Copy link
Author

log

@IcedTea2K
Copy link
Contributor

I'm able to repro the error. If I changed POSTGRES_PASSWORD in .env only, running docker compose up -d won't start up docker, since supabase_analytics will fail with the password authentication failed for user "supabase_admin" error above. This would prevent me from accessing the dashboard and other services, unless I remove the database rm -rf volumes/db/data/. It makes sense because the database postgres is created supabase_admin having the initial password. Trying to change password in .env won't change the role's password in the database.

So, I tried changing supabase_admin password directly (as well as postgres role). And then, I changed the password in .env. This will allow you to run supabase somewhat normally.

supabase/docker
⇡3% ➜ docker compose up -d

supabase/docker
⇡4% ➜ psql -h 127.0.0.1 -p 5432 -d postgres -U supabase_admin
Password for user supabase_admin:
psql (16.2 (Homebrew), server 15.1 (Ubuntu 15.1-1.pgdg20.04 1))
Type "help" for help.

postgres=# alter user supabase_admin with password 'changed_password';
ALTER ROLE
postgres=# alter user postgres with password 'changed_password';
ALTER ROLE
postgres=# \q

supabase/docker
⇡7% ➜ docker compose down


supabase/docker
⇡7% ➜ docker compose up

The reason why this only "somewhat" works is because the error still appears in the supabase_anlytics logs. But, it appears less (though, still frequent), and you could access other services (at least the dashboard since that's the only thing i tested). I've yet to find a way to cleanly change password :(

@InfoSecMastermind
Copy link
Author

It is more likely a bug.

@encima
Copy link
Member

encima commented Apr 15, 2024

Thanks for this @IcedTea2K ! This is a pretty clean guide and could be integrated into the docs for sure!

As I see it, I think this issue highlights 2 things:

  1. This is a non-trivial action and should be documented in more detail
  2. The steps outlined above could be rolled up into a CLI command to allow easy resetting of the environment (not just passwords but volumes as well)

Is that right or do you all see it differently?

@InfoSecMastermind
Copy link
Author

@encima In my opinion, The issue is that the above commands are also not suitable for production. They still contain errors. Putting them as a workaround in documentation will not be viable. They should also not be rolled up into a CLI command, including resetting volumes, because volumes would be better if they remained persistent, and that will only make it more broken and complex. Rather, the program needs to be fixed to handle changing passwords.

@IcedTea2K
Copy link
Contributor

@InfoSecMastermind I agree that the current method I'm using is not entirely ideal yet, since the error still lingers (debugging to find a better way rn). But I wouldn't conclude that it's a bug yet, because docker takes the password in the .env to set database password initially, and then uses it to log into the database subsequently. For example, this is how supabase_analytics "logging" into the postgres database.

POSTGRES_BACKEND_URL: postgresql://supabase_admin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}

So, changing the password in the environment variable only will not be picked up automatically, and result in invalid password. We need to reset the password for the database if we want to keep it or rm -rf volumes/db/data/ if we don't.

@encima I totally agree with your highlights. Once we find a clean way to reset password, documenting it and adding a CLI command will allow the users to do it both manually or automatically. And I'd assume for the CLI command, it'd be part of the supabase CLI right? I'd love to contribute to that

@IcedTea2K
Copy link
Contributor

Seems like I have to reset the password for the following users:

authenticator,dashboard_user,pgbouncer,postgres,supabase_admin,supabase_auth_admin,supabase_functions_admin,supabase_read_only_user,supabase_replication_admin,supabase_storage_admin

But, I'm still running into the password authentication failed for user "supabase_admin" for db, and analytics (the rest of the services run fine).

Would anyone have any clue why? It's recurrent and continuous logs, and doesn't fail any thing. I could still connect to the database, dashboard, etc.

@IcedTea2K
Copy link
Contributor

The problem narrows down to just supabase_analytics. All the services are working fine, except for the analytics. From the logs, it tries to connect to the postgres database as supabase_admin over and over again, but it keeps failing. @encima, would you know what's making it to continuously reconnect like that? I'm trying to look around, but couldn't find it, so I can't really identify what's performing the authentications, and with what credentials.

@Ziinc
Copy link
Contributor

Ziinc commented Apr 22, 2024

It's a bug due to the stored pg url, there is currently no updating done on startup .
Thanks @encima for getting this on my radar.

@encima encima added bug Something isn't working logs labels Apr 22, 2024
@Ziinc
Copy link
Contributor

Ziinc commented Apr 22, 2024

Recommended fix for now if you don't want to remove your docker volume is to update the stored pg url in the backends table under the analytics schema

@IcedTea2K
Copy link
Contributor

Confirmed that it works as the current work around

UPDATE _analytics.source_backends
SET config = jsonb_set(config, '{url}', '"postgresql://supabase_admin:$new_passwd@db:5432/postgres"', 'false')
WHERE type='postgres';

@vvvyyynet
Copy link

vvvyyynet commented Apr 26, 2024

@Ziinc, @IcedTea2K can you please give some more advice on how to access the backends from CLI? For me, removing volumes/db/data/ worked just fine for now, however later I may rely on your workaround. I'm very new to docker and supabase. Thanks!

@encima
Copy link
Member

encima commented Apr 26, 2024

@vvvyyynet I would wait until there are docs for this if you are not super comfortable as it can break things.

However, running the command above when connected to the database (i.e. from the SQL Editor or using psql) will work

@IcedTea2K
Copy link
Contributor

IcedTea2K commented Apr 26, 2024

@vvvyyynet If you'd like, here's a little script that I used to work around the bug. Just put it into a bash file (e.g., change_password.sh) and run ./change_password.sh new_pass, and everything should be automated for you.

#!/bin/bash

old_passwd=`grep POSTGRES_PASSWORD= .env | sed "s/.*=\(.*\)/\1/"`
if [ -z $1 ]; then
    echo "Please specify a new password"
    exit 1
fi
new_passwd=$1

PGPASSWORD=$old_passwd psql -h 127.0.0.1 -p 5432 -d postgres -U supabase_admin << EOT
    alter user anon with password '$new_passwd';
    alter user authenticated with password '$new_passwd';
    alter user authenticator with password '$new_passwd';
    alter user dashboard_user with password '$new_passwd';
    alter user pgbouncer with password '$new_passwd';
    alter user pgsodium_keyholder with password '$new_passwd';
    alter user pgsodium_keyiduser with password '$new_passwd';
    alter user pgsodium_keymaker with password '$new_passwd';
    alter user postgres with password '$new_passwd';
    alter user service_role with password '$new_passwd';
    alter user supabase_admin with password '$new_passwd';
    alter user supabase_auth_admin with password '$new_passwd';
    alter user supabase_functions_admin with password '$new_passwd';
    alter user supabase_read_only_user with password '$new_passwd';
    alter user supabase_replication_admin with password '$new_passwd';
    alter user supabase_storage_admin with password '$new_passwd';

    UPDATE _analytics.source_backends
    SET config = jsonb_set(config, '{url}', '"postgresql://supabase_admin:$new_passwd@db:5432/postgres"', 'false')
    WHERE type='postgres';
EOT

if [ $? -eq 0 ]; then
    sed -i -e "s/POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$new_passwd/g" .env
fi

@IcedTea2K
Copy link
Contributor

@encima @Ziinc any updates on this?

@inian inian added analytics o11y and analytics and removed logs labels May 17, 2024
@Ziinc Ziinc removed the help wanted Extra attention is needed label May 20, 2024
@Ziinc
Copy link
Contributor

Ziinc commented May 20, 2024

I will update this thread once the bugfix is out. it will be looked at in the coming week or so.

@Ziinc
Copy link
Contributor

Ziinc commented Jun 12, 2024

This bug has been fixed in Logflare v1.7.2

CLI version will be updated shortly.

@encima
Copy link
Member

encima commented Jun 13, 2024

Leaving this open so we can document the process in the self-hosting section

@encima encima added enhancement New feature or request self-hosted Issues related to self hosting and removed bug Something isn't working analytics o11y and analytics labels Jun 13, 2024
@Ziinc
Copy link
Contributor

Ziinc commented Sep 9, 2024

JFYI: the pg config bug has been more solidly fixed in logflare v1.8.9, please use that version instead.

@jpampa-dev
Copy link

Cree un bash con nombre change_password.sh, con el siguiente contenido. CONTAINER_NAME, es el nombre del contenedor donde esta corriendo la base de datos

#!/bin/bash

# Obtener la contraseña actual del archivo .env
old_passwd=`grep POSTGRES_PASSWORD= .env | sed "s/.*=\(.*\)/\1/"`
if [ -z $1 ]; then
    echo "Por favor, especifica una nueva contraseña"
    exit 1
fi
new_passwd=$1

# Nombre del contenedor PostgreSQL (reemplázalo si tu contenedor tiene un nombre diferente)
CONTAINER_NAME="supabase-db"

# Ejecutar el comando psql dentro del contenedor de Docker sin TTY
docker exec -i $CONTAINER_NAME psql -h 127.0.0.1 -p 5432 -d postgres -U supabase_admin << EOT
    alter user anon with password '$new_passwd';
    alter user authenticated with password '$new_passwd';
    alter user authenticator with password '$new_passwd';
    alter user dashboard_user with password '$new_passwd';
    alter user pgbouncer with password '$new_passwd';
    alter user pgsodium_keyholder with password '$new_passwd';
    alter user pgsodium_keyiduser with password '$new_passwd';
    alter user pgsodium_keymaker with password '$new_passwd';
    alter user postgres with password '$new_passwd';
    alter user service_role with password '$new_passwd';
    alter user supabase_admin with password '$new_passwd';
    alter user supabase_auth_admin with password '$new_passwd';
    alter user supabase_functions_admin with password '$new_passwd';
    alter user supabase_read_only_user with password '$new_passwd';
    alter user supabase_replication_admin with password '$new_passwd';
    alter user supabase_storage_admin with password '$new_passwd';

    UPDATE _analytics.source_backends
    SET config = jsonb_set(config, '{url}', '"postgresql://supabase_admin:$new_passwd@db:5432/postgres"', 'false')
    WHERE type='postgres';
EOT

# Verificar si la ejecución fue exitosa
if [ $? -eq 0 ]; then
    sed -i -e "s/POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$new_passwd/g" .env
    echo "Contraseña cambiada exitosamente."
else
    echo "Hubo un error al cambiar la contraseña."
fi

Le otorgue los permisos necesarios :

chmod  x ./change_password.sh

Luego ejecutar el comando:

./change_password.sh NEW_PASSWORD

Codigo inspirado en @IcedTea2K

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request good first issue Good for newcomers self-hosted Issues related to self hosting
Projects
Status: No status
Development

No branches or pull requests

7 participants