Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for ZIM files #3700

Closed
suncloudsmoon opened this issue Jul 7, 2024 · 0 comments
Closed

feat: Add support for ZIM files #3700

suncloudsmoon opened this issue Jul 7, 2024 · 0 comments

Comments

@suncloudsmoon
Copy link

Is your feature request related to a problem? Please describe.
The issue with a lot of LLMs is that they hallucinate if you ask them about obscure topics.

Describe the solution you'd like
There has been plenty of research that highlights the benefits of using Wikipedia in RAG to reduce hallucinations and increase the truthfulness of LLMs. Since Wikipedia is distributed as a single ZIM file via the ZIM format, I propose that Open WebUI should implement functionality to use ZIM files for RAG. One possible implementation route is to use the python binding of libzim to search for titles or do full text search (supported in some ZIM files) and use that information for RAG.

Describe alternatives you've considered
An alternative is to use the web search capability in Open WebUI for RAG. The drawback of web search is that it doesn't work offline and it doesn't always have the best information on a given topic (potential for misinformation). Another alternative solution is to create a pipeline addon that implements the above functionality.

Additional context
N/A

@suncloudsmoon suncloudsmoon changed the title enh: Add support for ZIM files feat: Add support for ZIM files Jul 7, 2024
@open-webui open-webui locked and limited conversation to collaborators Jul 8, 2024
@tjbck tjbck converted this issue into discussion #3711 Jul 8, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant