New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Domain whitelisting for web search results #3112

Merged

tjbck merged 14 commits into open-webui:dev from que-nguyen:searxng

Jun 17, 2024

Contributor

que-nguyen commented Jun 13, 2024

Note: Meta search engines like SearxNG do not support filtering results by domain natively, so this filtering is applied post-search.
Added a filter to restrict search results to specified domains.
Updated the processing of search results to apply the whitelist filter before returning the final results.
Ensured that only results from allowed domains (if specified in the whitelist) are included in the output.


Implement domain whitelisting for web search results

7b5f434

programminginhackerway commented Jun 14, 2024

SearxNG

Thanks, does it work well? I found that SearxNG only supports the high priority option and it can't handle the unwanted websites in results.


Merge branch 'dev' into searxng

a02ba52

Contributor Author

que-nguyen commented Jun 15, 2024

SearxNG

Thanks, does it work well? I found that SearxNG only supports the high priority option and it can't handle the unwanted websites in results.

Yes, it works well and addresses the issue by ensuring that only trusted websites are included and are under control.

tjbck reviewed

View reviewed changes

backend/apps/rag/main.py Outdated Show resolved Hide resolved

tjbck reviewed

View reviewed changes

backend/config.py Outdated Show resolved Hide resolved

tjbck reviewed

View reviewed changes

backend/config.py Outdated Show resolved Hide resolved

tjbck reviewed

View reviewed changes

backend/apps/rag/search/main.py Outdated Show resolved Hide resolved

tjbck reviewed

View reviewed changes

backend/apps/rag/search/brave.py Outdated Show resolved Hide resolved

Contributor

tjbck commented Jun 17, 2024

The rest LGTM!

tjbck and others added 12 commits

June 16, 2024 22:52


Merge pull request open-webui#3159 from open-webui/dev

9e4dd4b

0.3.5


Merge branch 'open-webui:main' into searxng

75e51ec


Refactor main.py

a3ac9ee

Rename RAG_WEB_SEARCH_WHITE_LIST_DOMAINS to RAG_WEB_SEARCH_DOMAIN_FILTER_LIST


Refactored config.py

b3d136b

Renamed RAG_WEB_SEARCH_WHITE_LIST_DOMAINS to RAG_WEB_SEARCH_DOMAIN_FILTER_LIST


Set filter_list as optional param in brave.py

a02139b


Set filter_list as optional param in duckduckgo.py

7d2ad8c


Set filter_list as optional param in google_pse.py

d8beed1


Refactor rag/main.py

3cc0e3e

Renamed function get_filtered_results


Set filter_list as optional param in searxng.py

9c446d9


Set filter_list as optional param in serper.py

6b8290f


Set filter_list as optional param in serply.py

bcb8423


Set filter_list as optional param in serpstack.py

c487385

Contributor

tjbck commented Jun 17, 2024

LGTM, Thanks!

tjbck merged commit 20f052e into open-webui:dev

3 of 5 checks passed

que-nguyen deleted the searxng branch

June 19, 2024 11:20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment