Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor inference engines to extends AIEngine #2347

Merged
merged 6 commits into from
Mar 22, 2024

Conversation

louis-jan
Copy link
Contributor

@louis-jan louis-jan commented Mar 14, 2024

Describe Your Changes

With the introduction of AIEngine, inference extensions can now extend the corresponding AIEngine to reduce the complexity of their code base

classDiagram
    AIEngine <|-- OAIEngine
    OAIEngine <|-- LocalOAIEngine
    LocalOAIEngine <|-- NitroExtension 
    LocalOAIEngine <|-- TensorRTLLMExtension
    OAIEngine <|--RemoteOAIEngine
    RemoteOAIEngine <|-- OpenAIExtension
    RemoteOAIEngine <|-- GroqAIExtension
    RemoteOAIEngine <|-- TritonTensorRTExtension 

class AIEngine {
<<Abstract>>
 string provider
 models()
 prePopulateModels()
}

class OAIEngine {
<<Abstract>>
 string inferenceUrl
 Model loadedModel
 inference()
 stopInference()
 headers()
}

class RemoteOAIEngine {
<<Abstract>>
 string apiKey
 headers()
}

class LocalOAIEngine {
<<abstract>>
 loadModel()
 unloadModel()
}

class OpenAIExtension {
 string inferenceUrl
 string apiKey
}
class GroqAIExtension{
 string inferenceUrl
 string apiKey
}
class TritonTensorRTExtension {
 string inferenceUrl
 string apiKey
}
class NitroExtension {
 spawnNitroProcess()
 loadModel()
 unloadModel()
 killSubprocess()
}
class TensorRTLLMExtension {
 spawnNitroProcess()
 loadModel()
 unloadModel()
 killSubprocess()
}
Loading

Fixes Issues

Self Checklist

  • Added relevant comments, esp in complex areas
  • Updated docs (for bug fixes / features)
  • Created issues for follow-up changes or refactoring needed

@github-actions github-actions bot added the type: bug Something isn't working label Mar 14, 2024
@louis-jan louis-jan force-pushed the chore/refactor-nitro-to-extends-localoaiengine branch 2 times, most recently from 77f9b05 to f611a53 Compare March 14, 2024 12:28
@louis-jan louis-jan changed the title fix: refactor nitro to extends localoaiengine fix: refactor nitro to extends LocalOAIEngine Mar 14, 2024
@louis-jan louis-jan force-pushed the chore/refactor-nitro-to-extends-localoaiengine branch from 5797a42 to a1a5836 Compare March 14, 2024 17:45
@louis-jan louis-jan marked this pull request as draft March 15, 2024 05:55
@louis-jan louis-jan marked this pull request as ready for review March 15, 2024 15:07
@louis-jan louis-jan force-pushed the chore/refactor-nitro-to-extends-localoaiengine branch 4 times, most recently from 60a1d44 to b21387c Compare March 20, 2024 05:45
@louis-jan louis-jan force-pushed the chore/refactor-nitro-to-extends-localoaiengine branch from b21387c to 75d49d0 Compare March 21, 2024 10:13
@louis-jan louis-jan force-pushed the chore/refactor-nitro-to-extends-localoaiengine branch from 0f526f0 to 8966f6f Compare March 21, 2024 11:38
@louis-jan louis-jan force-pushed the chore/refactor-nitro-to-extends-localoaiengine branch from 245d234 to 12c1c76 Compare March 21, 2024 14:43
@louis-jan louis-jan changed the title fix: refactor nitro to extends LocalOAIEngine fix: refactor inference engines to extends AIEngine Mar 21, 2024
@louis-jan louis-jan mentioned this pull request Mar 21, 2024
28 tasks
@louis-jan louis-jan merged commit acbec78 into dev Mar 22, 2024
7 checks passed
@louis-jan louis-jan deleted the chore/refactor-nitro-to-extends-localoaiengine branch March 22, 2024 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants