feat: External databases: reconnect to postgresql & mysql when connection is broken #2666

PeterBeklemishev · 2024-05-30T12:00:14Z

Pull Request Checklist

Note to first-time contributors: Please open a discussion post in Discussions and describe your changes before submitting a pull request.

Before submitting, make sure you've checked the following:

Target branch: Please verify that the pull request targets the dev branch.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Testing: Have you written and run sufficient tests for validating the changes?
Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
Label: To cleary categorize this pull request, assign a relevant label to the pull request title, using one of the following:
- BREAKING CHANGE: Significant changes that may affect compatibility
- build: Changes that affect the build system or external dependencies
- ci: Changes to our continuous integration processes or workflows
- chore: Refactor, cleanup, or other non-functional code changes
- docs: Documentation update or addition
- feat: Introduces a new feature or enhancement to the codebase
- fix: Bug fix or error correction
- i18n: Internationalization or localization changes
- perf: Performance improvement
- refactor: Code restructuring for better maintainability, readability, or scalability
- style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
- test: Adding missing tests or correcting existing tests
- WIP: Work in progress, a temporary label for incomplete or ongoing work

Changelog Entry

Added

Auto-reconnection to postgresql/mysql databases, proposed in enh: more mature support for external (non-SQLite) databases #2660

Additional Information

Based on #2660

Example exception occuring without added reconnect mixin in case of closed connection

ollama        | [GIN] 2024/05/30 - 12:26:40 | 200 |      66.969µs |      172.28.0.4 | GET      "/api/tags"
open-webui    | INFO:     127.0.0.1:43728 - "GET /health HTTP/1.1" 500 Internal Server Error
open-webui    | ERROR:    Exception in ASGI application
open-webui    |     Exception Group Traceback (most recent call last):
open-webui    |   |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
open-webui    |   |     yield
open-webui    |   |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 190, in __call__
open-webui    |   |     async with anyio.create_task_group() as task_group:
open-webui    |   |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
open-webui    |   |     raise BaseExceptionGroup(
open-webui    |   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
open-webui    |    - ---------------- 1 ----------------
open-webui    |     | Traceback (most recent call last):
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
open-webui    |     |     result = await app(  # type: ignore[func-returns-value]
open-webui    |     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
open-webui    |     |     return await self.app(scope, receive, send)
open-webui    |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
open-webui    |     |     await super().__call__(scope, receive, send)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
open-webui    |     |     await self.middleware_stack(scope, receive, send)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
open-webui    |     |     raise exc
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
open-webui    |     |     await self.app(scope, receive, _send)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
open-webui    |     |     with collapse_excgroups():
open-webui    |     |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
open-webui    |     |     self.gen.throw(typ, value, traceback)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
open-webui    |     |     raise exc
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
open-webui    |     |     response = await self.dispatch_func(request, call_next)
open-webui    |     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/app/backend/main.py", line 365, in update_embedding_function
open-webui    |     |     response = await call_next(request)
open-webui    |     |                ^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
open-webui    |     |     raise app_exc
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
open-webui    |     |     await self.app(scope, receive_or_disconnect, send_no_error)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
open-webui    |     |     with collapse_excgroups():
open-webui    |     |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
open-webui    |     |     self.gen.throw(typ, value, traceback)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
open-webui    |     |     raise exc
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
open-webui    |     |     response = await self.dispatch_func(request, call_next)
open-webui    |     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/app/backend/main.py", line 351, in check_url
open-webui    |     |     await get_all_models()
open-webui    |     |   File "/app/backend/main.py", line 408, in get_all_models
open-webui    |     |     custom_models = Models.get_all_models()
open-webui    |     |                     ^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/app/backend/apps/webui/models/models.py", line 148, in get_all_models
open-webui    |     |     return [ModelModel(**model_to_dict(model)) for model in Model.select()]
open-webui    |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 7260, in __iter__
open-webui    |     |     self.execute()
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 2025, in inner
open-webui    |     |     return method(self, database, *args, **kwargs)
open-webui    |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 2096, in execute
open-webui    |     |     return self._execute(database)
open-webui    |     |            ^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 2269, in _execute
open-webui    |     |     cursor = database.execute(self)
open-webui    |     |              ^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3319, in execute
open-webui    |     |     return self.execute_sql(sql, params)
open-webui    |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3309, in execute_sql
open-webui    |     |     with __exception_wrapper__:
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3077, in __exit__
open-webui    |     |     reraise(new_type, new_type(exc_value, *exc_args), traceback)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 196, in reraise
open-webui    |     |     raise value.with_traceback(tb)
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3310, in execute_sql
open-webui    |     |     cursor = self.cursor()
open-webui    |     |              ^^^^^^^^^^^^^
open-webui    |     |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3303, in cursor
open-webui    |     |     return self._state.conn.cursor()
open-webui    |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |     | peewee.InterfaceError: connection already closed
open-webui    |      ------------------------------------
open-webui    | 
open-webui    | During handling of the above exception, another exception occurred:
open-webui    | 
open-webui    | Traceback (most recent call last):
open-webui    |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
open-webui    |     result = await app(  # type: ignore[func-returns-value]
open-webui    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
open-webui    |     return await self.app(scope, receive, send)
open-webui    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
open-webui    |     await super().__call__(scope, receive, send)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
open-webui    |     await self.middleware_stack(scope, receive, send)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
open-webui    |     raise exc
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
open-webui    |     await self.app(scope, receive, _send)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
open-webui    |     with collapse_excgroups():
open-webui    |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
open-webui    |     self.gen.throw(typ, value, traceback)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
open-webui    |     raise exc
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
open-webui    |     response = await self.dispatch_func(request, call_next)
open-webui    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/app/backend/main.py", line 365, in update_embedding_function
open-webui    |     response = await call_next(request)
open-webui    |                ^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
open-webui    |     raise app_exc
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
open-webui    |     await self.app(scope, receive_or_disconnect, send_no_error)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
open-webui    |     with collapse_excgroups():
open-webui    |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
open-webui    |     self.gen.throw(typ, value, traceback)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
open-webui    |     raise exc
open-webui    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
open-webui    |     response = await self.dispatch_func(request, call_next)
open-webui    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/app/backend/main.py", line 351, in check_url
open-webui    |     await get_all_models()
open-webui    |   File "/app/backend/main.py", line 408, in get_all_models
open-webui    |     custom_models = Models.get_all_models()
open-webui    |                     ^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/app/backend/apps/webui/models/models.py", line 148, in get_all_models
open-webui    |     return [ModelModel(**model_to_dict(model)) for model in Model.select()]
open-webui    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 7260, in __iter__
open-webui    |     self.execute()
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 2025, in inner
open-webui    |     return method(self, database, *args, **kwargs)
open-webui    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 2096, in execute
open-webui    |     return self._execute(database)
open-webui    |            ^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 2269, in _execute
open-webui    |     cursor = database.execute(self)
open-webui    |              ^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3319, in execute
open-webui    |     return self.execute_sql(sql, params)
open-webui    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3309, in execute_sql
open-webui    |     with __exception_wrapper__:
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3077, in __exit__
open-webui    |     reraise(new_type, new_type(exc_value, *exc_args), traceback)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 196, in reraise
open-webui    |     raise value.with_traceback(tb)
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3310, in execute_sql
open-webui    |     cursor = self.cursor()
open-webui    |              ^^^^^^^^^^^^^
open-webui    |   File "/usr/local/lib/python3.11/site-packages/peewee.py", line 3303, in cursor
open-webui    |     return self._state.conn.cursor()
open-webui    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
open-webui    | peewee.InterfaceError: connection already closed

Screenshots or Videos

none

…nnected

tjbck · 2024-05-30T18:11:26Z

backend/apps/webui/internal/wrappers.py

+
+
+def register_peewee_databases():
+ register_database(MySQLDatabase, 'mysql')


mysql isn't officially supported, this should be removed.

tjbck · 2024-05-30T18:11:53Z

More testing wanted here but the rest LGTM!

tjbck · 2024-06-01T17:22:48Z

backend/apps/webui/internal/wrappers.py

+ pass
+
+
+def register_peewee_databases():


perhaps we should also add SQLiteReconnectMixin here? were you able to test this working using sqlite as well?

@PeterBeklemishev

Currently I don't have any ideas how to reproduce disconnection from sqlite :(

Disconnection from pg can be easily triggered, so I added this to github action, can someone trigger them via approving workflow in this PR?

justinh-rahb · 2024-06-03T15:20:36Z

Any more work needed here? I think it's very important that we get this merged.

tjbck · 2024-06-03T19:57:31Z

I just need to verify it won't break the sqlite connections with 100% confidence then it's good to merge 👍

justinh-rahb · 2024-06-05T20:01:08Z

I just need to verify it won't break the sqlite connections with 100% confidence then it's good to merge 👍

Yaaa... about that. I have not gotten this PR to work on a SQLite database thus far.

matthieucharreire · 2024-06-11T13:06:32Z

I just need to verify it won't break the sqlite connections with 100% confidence then it's good to merge 👍

Yaaa... about that. I have not gotten this PR to work on a SQLite database thus far.

@justinh-rahb are you stuck on this issue ? Do you need help on the matter, or is it work in progress ?

I agree with you that this improvement is very important 🙂

nitinkr0411 · 2024-06-13T21:26:44Z

Not sure if this is the right place, but should we be passing DATABASE_URL.
I think it would be useful if we can pass it in pieces as separate vars

DATABASE_USER
DATABASE_PASS
DATABASE_HOST

This would be useful in CI scenarios and/or deploying via AWS CloudFormation where:

DATABASE_USER -> Saved in SecretsManager
DATABASE_PASS -> Saved in SecretsManager / Time-Rotated Credentials
DATABASE_HOST -> Generated from cloudformation created RDS resource

PeterBeklemishev · 2024-06-14T13:23:33Z

Not sure if this is the right place, but should we be passing DATABASE_URL. I think it would be useful if we can pass it in pieces as separate vars

DATABASE_USER

DATABASE_PASS

DATABASE_HOST

This would be useful in CI scenarios and/or deploying via AWS CloudFormation where:

DATABASE_USER -> Saved in SecretsManager

DATABASE_PASS -> Saved in SecretsManager / Time-Rotated Credentials

DATABASE_HOST -> Generated from cloudformation created RDS resource

hi!
@rionitinkumar this most likely should be discussed as separate issue/pull request

nitinkr0411 · 2024-06-14T17:35:24Z

Not sure if this is the right place, but should we be passing DATABASE_URL. I think it would be useful if we can pass it in pieces as separate vars

DATABASE_USER

DATABASE_PASS

DATABASE_HOST

This would be useful in CI scenarios and/or deploying via AWS CloudFormation where:

DATABASE_USER -> Saved in SecretsManager

DATABASE_PASS -> Saved in SecretsManager / Time-Rotated Credentials

DATABASE_HOST -> Generated from cloudformation created RDS resource

hi! @rionitinkumar this most likely should be discussed as separate issue/pull request

Created issue #3179

tjbck · 2024-06-15T09:45:22Z

This won't get merged until we can confirm with 100% certainty that the sqlite issue gets fixed, help wanted here 🙌

TSCarterJr · 2024-06-24T22:24:25Z

In the past, the only ways i've been able to test severing the connection, are to either code up a test function to disconnect from the manager, or, the easier approach:

rename the sqlite file, then wait a few seconds, and rename it back.

PeterBeklemishev changed the title ~~External databases: reconnect to postgresql & mysql when connection is broken~~ WIP: External databases: reconnect to postgresql & mysql when connection is broken May 30, 2024

PeterBeklemishev changed the base branch from main to dev May 30, 2024 12:03

PeterBeklemishev changed the title ~~WIP: External databases: reconnect to postgresql & mysql when connection is broken~~ feat: External databases: reconnect to postgresql & mysql when connection is broken May 30, 2024

PeterBeklemishev added 2 commits May 30, 2024 20:41

Reconnect to postgresql & mysql external databases when getting disco…

dfbc125

…nnected

Test reconnection to postgres in gh actions

e1fa453

PeterBeklemishev force-pushed the feature-external-db-reconnect branch from 6a10c0c to e1fa453 Compare May 30, 2024 13:42

Fix rebase artifacts

e59e1f5

PeterBeklemishev marked this pull request as ready for review May 30, 2024 13:45

PeterBeklemishev mentioned this pull request May 30, 2024

enh: more mature support for external (non-SQLite) databases #2660

Closed

tjbck reviewed May 30, 2024

View reviewed changes

Drop mysql restarts

ad32a2e

tjbck reviewed Jun 1, 2024

View reviewed changes

tjbck added bug Something isn't working help wanted Extra attention is needed core core feature labels Jun 16, 2024

perfectra1n mentioned this pull request Jun 16, 2024

feat: external db reconnect #3221

Merged

8 tasks

tjbck merged commit 75d7130 into open-webui:dev Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: External databases: reconnect to postgresql & mysql when connection is broken #2666

feat: External databases: reconnect to postgresql & mysql when connection is broken #2666

PeterBeklemishev commented May 30, 2024 •

edited

Loading

tjbck May 30, 2024

PeterBeklemishev May 31, 2024

tjbck commented May 30, 2024

tjbck Jun 1, 2024

perfectra1n Jun 4, 2024

PeterBeklemishev Jun 4, 2024

justinh-rahb commented Jun 3, 2024

tjbck commented Jun 3, 2024

justinh-rahb commented Jun 5, 2024

matthieucharreire commented Jun 11, 2024

nitinkr0411 commented Jun 13, 2024 •

edited

Loading

PeterBeklemishev commented Jun 14, 2024 •

edited

Loading

nitinkr0411 commented Jun 14, 2024

tjbck commented Jun 15, 2024

TSCarterJr commented Jun 24, 2024



		def register_peewee_databases():
		register_database(MySQLDatabase, 'mysql')

feat: External databases: reconnect to postgresql & mysql when connection is broken #2666

feat: External databases: reconnect to postgresql & mysql when connection is broken #2666

Conversation

PeterBeklemishev commented May 30, 2024 • edited Loading

Pull Request Checklist

Note to first-time contributors: Please open a discussion post in Discussions and describe your changes before submitting a pull request.

Changelog Entry

Added

Additional Information

Screenshots or Videos

tjbck May 30, 2024

Choose a reason for hiding this comment

PeterBeklemishev May 31, 2024

Choose a reason for hiding this comment

tjbck commented May 30, 2024

tjbck Jun 1, 2024

Choose a reason for hiding this comment

perfectra1n Jun 4, 2024

Choose a reason for hiding this comment

PeterBeklemishev Jun 4, 2024

Choose a reason for hiding this comment

justinh-rahb commented Jun 3, 2024

tjbck commented Jun 3, 2024

justinh-rahb commented Jun 5, 2024

matthieucharreire commented Jun 11, 2024

nitinkr0411 commented Jun 13, 2024 • edited Loading

PeterBeklemishev commented Jun 14, 2024 • edited Loading

nitinkr0411 commented Jun 14, 2024

tjbck commented Jun 15, 2024

TSCarterJr commented Jun 24, 2024

PeterBeklemishev commented May 30, 2024 •

edited

Loading

nitinkr0411 commented Jun 13, 2024 •

edited

Loading

PeterBeklemishev commented Jun 14, 2024 •

edited

Loading