FWIW blocking the two connections didn't appear to affect the functionality of Uploading a document. I was later able to select it and use it in context with #, so I'm really confused as to why those connections are even necessary at all 🤔

Hi, Thanks for reporting this issue. Could you verify that AWS ELB connection is 100% occurring from the webui-side? Our backend code does no contain any code that explicitly makes connection with AWS ELB, so my guess is the request is made from one of our dependency libraries. If you could narrow down what part of the code making the connection, that would be tremendously helpful, Thanks!

Yup makes sense!

I'll try to narrow this down 👌 As you said, If you're not doing this explicitly in this codebase then I consider a sneaky supply chain type of thing 🤣

So here we go:

Text version(s):

Docker wants to connect to a046be49099ce4659abbcfa853797f20-5fd7cc9498e4883e.elb.ap-southeast-1.amazonaws.com on TCP port 443 (https)

Docker wants to connect to packages.unstructured.io on TCP port 443 (https)

Screenshots:

Note that this is the container itself trying to do this, so something to do with the backend.

Doing a search for the 2nd connection yield this:

https://github.com/Unstructured-IO/unstructured/blob/d11c70cf83fdb8a08fed2cf01c6c0bd114d817df/unstructured/utils.py#L287-L319

Are we using this in the backenda anywhere? 🤔

Here's a list of our suspects:

langchain
langchain-community
chromadb
sentence_transformers
pypdf
docx2txt
unstructured
markdown
pypandoc
pandas
openpyxl
pyxlsb
xlrd

We are:

https://github.com/ollama-webui/ollama-webui/blob/cb5520c519dde81bfe08ce358753ab7f11417f97/backend/requirements.txt#L25

Why does it need to connect to an external service? 🤔

I can't figure out this random ELB though, might need some help figuring that one out. But at least we have some culprits now.... The question is, what do we do about it? Blocking both doesn't adversely affect Ollama Web UI in any way that I can tell hmmm

Oh wow!

def scarf_analytics():
...

If this library is sending analytics, that's disgusting 😱

UnstructuredMarkdownLoader seems to be the culprit, investigating more.

Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.

www.unstructured.io/

I have half a mind to go yell at this company and ask them to please explain themselves 🤣 Shame on them!

Just reviewed the code, I reckon setting DO_NOT_TRACK env var to True will stop the telemetry, could you try testing it?

Love it! Let's do it, happy to test the fix 👌

And thank you for responding to this so quickly! When you're self hosting and insisting on doing things locally, you really don't expect your software to reach out to the internet without you knowing about it 😅

Some kudos I posted for you 😅

Good find guys, ya that definitely not nice of them to do. Is there any disclosure from the libary anywhere?

Good find guys, ya that definitely not nice of them to do. Is there any disclosure from the libary anywhere?

Are you suggesting we file a bug upstream too? It was a bit of a rude surprise to be honest 😅

@justinh-rahb none I can find from their readme :/

EDIT: they do mention at the very bottom of their readme to set the environment variable SCARF_NO_ANALYTICS=true.

Added

ENV SCARF_NO_ANALYTICS true
ENV DO_NOT_TRACK true

with #694, it should disable the telemetry. Please try it out and let me know!

With RAG being as hot as it is right now, I guess we shouldn't be surprised that some libary authors are cashing in on the user data flowing through their code.

Perhaps it'll be prudent to think about dependency audits in the future. With Ollama now supporting a broad range of CPU-only configurations, it can be integrated into GitHub Actions, along with Ollama-WebUI for thorough end-to-end testing. I'm going to give this a think over the weekend, I seem to recall there being a thread in discussions about using the webUI API directly that may come in handy here, time to do some research...

@prologic has the issue been resolved with the latest release?

I pulled the latest Docker image and restarted my local instance and so far so good 😊

I'll close this issue for now, feel free to open new issues if you encounter any spywares from the dependency supply chain, thanks!

Do you have an example code how to upload a document programatically through an api? is it possible