Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: set PostgreSQL SSL options in django #4550

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

fix: set PostgreSQL SSL options in django #4550

wants to merge 1 commit into from

Conversation

rupert
Copy link

@rupert rupert commented Jul 20, 2023 •
edited
Loading

We're attempting to deploy Label Studio to GCP using the official Helm chart but it's unable to connect to Cloud SQL with SSL.

These are the values we're using:

global:
  pgConfig:
    host: "1.2.3.4"
    port: 5432
    userName: "label-studio"
    dbName: "label-studio"
    password:
      secretName: "postgres-password"
      secretKey: "postgres-password"
    ssl:
      pgSslMode: "verify-ca"
      pgSslSecretName: "postgres-tls"
      pgSslRootCertSecretKey: "server-ca.pem"
      pgSslCertSecretKey: "client-cert.pem"
      pgSslKeySecretKey: "client-key.pem"

postgresql:
  enabled: false

redis:
  enabled: false

(replaced the real IP with 1.2.3.4)

These are the logs from the k8s pod:

./deploy/docker-entrypoint.sh: Looking for init scripts in /label-studio/deploy/docker-entrypoint.d/app/
./deploy/docker-entrypoint.sh: Launching /label-studio/deploy/docker-entrypoint.d/app/20-wait-for-db.sh
=> Waiting for postgres...
=> Postgres is up.
./deploy/docker-entrypoint.sh: Launching /label-studio/deploy/docker-entrypoint.d/app/30-run-db-migrations.sh
=> Skipping run db migrations.
./deploy/docker-entrypoint.sh: Launching /label-studio/deploy/docker-entrypoint.d/app/31-run-label-studio-init.sh
=> Run label-studio init...
=> Hostname correctly is set to: <snip>
=> Database and media directory: /label-studio/data
=> Static URL is set to: /static/
=> Hostname correctly is set to: <snip>
=> Database and media directory: /label-studio/data
=> Static URL is set to: /static/
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /pypi/label-studio/json HTTP/1.1" 200 57083
Can't read version file: ls-version_.py. Fall back to: version_.py
[2023-07-20 16:07:13,694] [root::_read_py::61] [WARNING] Can't read version file: ls-version_.py. Fall back to: version_.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/django/__init__.py", line 622, in connect
    return real_connect(self)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "1.2.3.4", port 5432 failed: FATAL:  connection requires a valid client certificate
connection to server at "1.2.3.4", port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "1.2.3.4", user "postgres", database "label-studio", SSL off


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/label-studio", line 33, in <module>
    sys.exit(load_entry_point('label-studio', 'console_scripts', 'label-studio')())
  File "/label-studio/label_studio/server.py", line 297, in main
    _apply_database_migrations()
  File "/label-studio/label_studio/server.py", line 65, in _apply_database_migrations
    if not is_database_synchronized(DEFAULT_DB_ALIAS):
  File "/label-studio/label_studio/server.py", line 58, in is_database_synchronized
    executor = MigrationExecutor(connection)
  File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/loader.py", line 220, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/recorder.py", line 55, in has_table
    with self.connection.cursor() as cursor:
  File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 218, in ensure_connection
    with self.wrap_database_errors:
  File "/usr/local/lib/python3.10/dist-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.10/dist-packages/sentry_sdk/integrations/django/__init__.py", line 622, in connect
    return real_connect(self)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "1.2.3.4", port 5432 failed: FATAL:  connection requires a valid client certificate
connection to server at "1.2.3.4", port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "1.2.3.4", user "postgres", database "label-studio", SSL off

Sentry is attempting to send 2 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit

When the pod runs the wait for db script on startup it's able to connect to the database successfully with SSL:

./deploy/docker-entrypoint.sh: Launching /label-studio/deploy/docker-entrypoint.d/app/20-wait-for-db.sh
=> Waiting for postgres...
=> Postgres is up.

However when it continues to start the django app we get an error:

django.db.utils.OperationalError: connection to server at "1.2.3.4", port 5432 failed: FATAL:  connection requires a valid client certificate
connection to server at "1.2.3.4", port 5432 failed: FATAL:  pg_hba.conf rejects connection for host "1.2.3.4", user "postgres", database "label-studio", SSL off

It seems django is trying to connect to the database without SSL which is blocked by our Cloud SQL configuration because it requires a client certificate.

Looking at the django settings it seems the app is not using the POSTGRE_* environment variables to set the SSL options when connecting to PostgreSQL. This is a little surprising as there's documentation etc. for this feature, so perhaps there's something we've missed?

This pull request updates the django settings to use those environment variables where available. I've tested this PR in our cluster and the django app is able to connect to the database, and the app starts successfully.

@netlify
Copy link

netlify bot commented Jul 20, 2023

👷 Deploy request for label-studio-docs-new-theme pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit df9c56b

@netlify
Copy link

netlify bot commented Jul 20, 2023

👷 Deploy request for heartex-docs pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit df9c56b

@github-actions github-actions bot added the fix label Jul 20, 2023
@rupert rupert marked this pull request as ready for review July 20, 2023 17:07
@erinmikailstaples
Copy link
Contributor

Hey @rupert — thank you so much for creating this! I really appreciate this!

I'll flag to our internal team as well :)

@erinmikailstaples erinmikailstaples added community:reviewed Issue has been reviewed by the Label Studio Community Team. community:fix labels Jul 24, 2023
@rupert
Copy link
Author

rupert commented Oct 4, 2023

@makseq @niklub would it be possible to review this PR when you have a chance?

@robot-ci-heartex
Copy link
Collaborator

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@sajarin sajarin added the Community Community Feature Requests, Open Issues, Bugs Reported, or Comments label Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@makseq makseq Awaiting requested review from makseq

@niklub niklub Awaiting requested review from niklub

@erinmikailstaples erinmikailstaples Awaiting requested review from erinmikailstaples

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
community:fix community:reviewed Issue has been reviewed by the Label Studio Community Team. Community Community Feature Requests, Open Issues, Bugs Reported, or Comments fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@rupert @erinmikailstaples @robot-ci-heartex @sajarin