-
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add explicit setup instructions for locally hosted AI #7497
Labels
story
Features perceivable by end-users
Comments
Issue Status: 🆕 *Untriaged*🆕 Untriaged The team has not yet reviewed the issue. We usually do it within one business day. This is an automatic reply by the bot. |
1 task
Great, I successfully connected! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
As I've documented in the discussion on the subject, it is actually relatively easy to redirect the openAI requests to a locally hosted solution (I've only tested with text-generation-webui but others should be usable). This information should be formally documented and the relevant code lines added.
I discuss my findings in detail in #7030 but I'll summarise the details here:
CMD_FLAGS.txt (This will default to listening on a global port 5000, consider additional security in production):
--api --listen --api-key "sk-111111111111111111111111111111111111111111111111"
Model Selection and Settings:
Any 8k context length model, tested with Llama-3-8B-Instruct-262k.Q5_K_M.gguf with a set context length of 8192
Character Selection:
Due to the None Character Error oobabooga/text-generation-webui#4320 bug, we need to manually add a new character to the text-generation-webui character page. Here is one that can be used, it just needs to match the "None" name as the context is overridden by AFFiNE. None.json
Use case
As self-hosting becomes more established and the kinks ironed out, I foresee a lot of demand for being able to run our own AI models locally. Not only is this critical for most internal business infrastructure, which AFFiNE and its competitors are very well suited for supporting, but it also allows for far greater customisation of how the software operates. Preferably, we'd also be able to directly modify the context string so we can set up our own branded AIs just as there are plans to allow us to modify the UI to suit our individual needs.
Anything else?
Original Discussion and Research: #7030
Text-Generation-WebUI's OPENAI API substitution documentation: https://github.com/oobabooga/text-generation-webui/wiki/12-‐-OpenAI-API
Text-Generation-WebUI Character Bug: oobabooga/text-generation-webui#4320
Text-Generation-WebUI "None" Character Workaround: https://github.com/user-attachments/files/16227649/None.json
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: