Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Multiple inference engines for nitro and openai #814

Merged
merged 58 commits into from
Dec 8, 2023
Merged

feat: Multiple inference engines for nitro and openai #814

merged 58 commits into from
Dec 8, 2023

Conversation

hiro-v
Copy link
Contributor

@hiro-v hiro-v commented Dec 1, 2023 •
edited
Loading

  • Refactor nitro inference extension
  • OAI compatible inference extension
  • fs.exists() abstraction
  • Route model init, model request
  • Refactor calling direct function to Even Based system

Ref:

@hiro-v hiro-v added the P1: important Important feature / fix label Dec 1, 2023
@hiro-v hiro-v requested a review from a team December 1, 2023 08:51
@hiro-v hiro-v self-assigned this Dec 1, 2023
@hiro-v hiro-v marked this pull request as draft December 1, 2023 08:51
@hiro-v hiro-v changed the title [WIP] Multiple inference engines for nitro and openai [WIP] Feature: Multiple inference engines for nitro and openai Dec 1, 2023
@0xSage 0xSage changed the title [WIP] Feature: Multiple inference engines for nitro and openai feat: Multiple inference engines for nitro and openai Dec 1, 2023
@0xSage
Copy link
Contributor

0xSage commented Dec 1, 2023

btw @vuonghoainam can just title issues feat: ... to trigger ci. WIP is implied since your PR is a draft

@0xSage 0xSage added the type: feature request A new feature label Dec 1, 2023
@hiro-v hiro-v mentioned this pull request Dec 1, 2023
Closed
5 tasks
@0xSage 0xSage added P0: critical Mission critical and removed P1: important Important feature / fix labels Dec 3, 2023
@hiro-v hiro-v mentioned this pull request Dec 4, 2023
Closed
@hiro-v hiro-v added this to the v0.4.0 milestone Dec 4, 2023
@hiro-v hiro-v marked this pull request as ready for review December 4, 2023 19:06
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 5, 2023 •
edited
Loading

Preview URL: https://48b9ddec.jan-qn8.pages.dev

@hiro-v
Copy link
Contributor Author

hiro-v commented Dec 6, 2023

How to use OpenAI and Nitro backed models
===> Once the app booted up, there are engines/*.json files
CleanShot 2023-12-06 at 09 25 51

OAI models

  1. Update base_url and api_key. By default this is OpenAI platform API but you can change it to any OAI compatible API out there (Azure, vLLM OAI entrypoint, etc)
    CleanShot 2023-12-06 at 09 26 05
  2. Go the Hub and search for OpenAI (I added 3 models there), click Use
    CleanShot 2023-12-06 at 09 25 18
  3. Create a thread and start to use
    CleanShot 2023-12-06 at 09 26 50

GGUF models (backed by engine=nitro

  1. Find any GGUF model in the hub
    CleanShot 2023-12-06 at 09 27 12
  2. Create a new thread or use in existing thread with GGUF model, then just use it
    CleanShot 2023-12-06 at 09 28 14

namchuai
namchuai reviewed Dec 7, 2023
View reviewed changes
core/src/fs.ts Outdated Show resolved Hide resolved
hiro-v and others added 20 commits December 8, 2023 23:06
@hiro-v
fix: Passing model object instead of model id
4266d86
@hiro-v
fix: spreading model.paramters object and update chatCompletion route
f34024a
@hiro-v
fix: Spreading model.parameters
a2cf42a
@hiro-v
fix: Add type def global for nitro extension - inference
44bfcaa
@hiro-v
feat: Add nitro inference engine stop model handler
3987fdc
@hiro-v
fix: debugging
6add24c
@hiro-v
fix: Fix resend button with model object
bb4b4c5
@hiro-v
fix: Add engine nitro
28368ee
@hiro-v
fix: Hub fix for undefined model size
6fe901b
@hiro-v
feat: Add openai models
2a853a2
@hiro-v
fix: Temporary disable model parameters spreading
5f3cf2b
@hiro-v
fix: Enforce openai inference to load settings from openai.json onLoa…
e1190ec 
…d and onModelInit
@hiro-v
chore: remove unused console.log
975e971
@hiro-v
fix: Add hack waiting for model loading
9daee14
@hiro-v
fix: model gpt json
6cd4cb9
@hiro-v
fix: Update lint for engine in ThreadAssistantInfo
fbf8ff9
@hiro-v
fix: Update code based on comments from @james
1177007
@louis-jan @hiro-v
chore: add ready state to remote models
ef9dfc9
@louis-jan @hiro-v
chore: stop inference event
b939692
@hiro-v
fix: Change base_url to full_url
f9e73b0
@hiro-v hiro-v merged commit ee16683 into main Dec 8, 2023
4 checks passed
@hiro-v hiro-v deleted the feat/inference_engines branch December 8, 2023 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@namchuai namchuai namchuai left review comments

@louis-jan louis-jan louis-jan approved these changes

Assignees

@hiro-v hiro-v

Labels
P0: critical Mission critical type: feature request A new feature
Projects
Jan Project
Archived in project
Milestone
0.4.0
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@hiro-v @0xSage @namchuai @louis-jan