You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The issue with a lot of LLMs is that they hallucinate if you ask them about obscure topics.
Describe the solution you'd like
There has been plenty of research that highlights the benefits of using Wikipedia in RAG to reduce hallucinations and increase the truthfulness of LLMs. Since Wikipedia is distributed as a single ZIM file via the ZIM format, I propose that Open WebUI should implement functionality to use ZIM files for RAG. One possible implementation route is to use the python binding of libzim to search for titles or do full text search (supported in some ZIM files) and use that information for RAG.
Describe alternatives you've considered
An alternative is to use the web search capability in Open WebUI for RAG. The drawback of web search is that it doesn't work offline and it doesn't always have the best information on a given topic (potential for misinformation). Another alternative solution is to create a pipeline addon that implements the above functionality.
Additional context
N/A
The text was updated successfully, but these errors were encountered:
suncloudsmoon
changed the title
enh: Add support for ZIM files
feat: Add support for ZIM files
Jul 7, 2024
Is your feature request related to a problem? Please describe.
The issue with a lot of LLMs is that they hallucinate if you ask them about obscure topics.
Describe the solution you'd like
There has been plenty of research that highlights the benefits of using Wikipedia in RAG to reduce hallucinations and increase the truthfulness of LLMs. Since Wikipedia is distributed as a single ZIM file via the ZIM format, I propose that Open WebUI should implement functionality to use ZIM files for RAG. One possible implementation route is to use the python binding of libzim to search for titles or do full text search (supported in some ZIM files) and use that information for RAG.
Describe alternatives you've considered
An alternative is to use the web search capability in Open WebUI for RAG. The drawback of web search is that it doesn't work offline and it doesn't always have the best information on a given topic (potential for misinformation). Another alternative solution is to create a pipeline addon that implements the above functionality.
Additional context
N/A
The text was updated successfully, but these errors were encountered: