Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Task Model to use the same LLM parameters as in Workspaces #3286

Closed
boshk0 opened this issue Jun 19, 2024 · 1 comment
Closed

Comments

@boshk0
Copy link

boshk0 commented Jun 19, 2024

Is your feature request related to a problem? Please describe.
Not really a problem, but rather an optimisation. I'm currently hosting Llama3 and would like to reuse this model for every task like User queries, Chat title generation, Web search. The issue I have is that I have set larger context length for the model in the Workspaces, which differs to the one (default) used in Task Model. The result is that Ollama constantly reloads the model, since the LLM parameters are different. If they are the same, the model will be kept in the GPU memory and will be much efficient.

Describe the solution you'd like
A best solultion would be to have an option to use the workspace model's parameters in Set Task Model, or less ideal to have a way to override the default LLM parameters using environment variables.

Describe alternatives you've considered
The only alternative I have found is to use very small model that can fit in the GPU memory along with the larger and use it as Task Model.

@boshk0 boshk0 changed the title Task Model to use the same LLM parameters as in Workspaces Feature request: Task Model to use the same LLM parameters as in Workspaces Jun 19, 2024
@boshk0 boshk0 closed this as completed Jun 19, 2024
@boshk0
Copy link
Author

boshk0 commented Jun 19, 2024

Duplicate:
#3106

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant