-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Non-streaming completion API #75
Comments
This is a very cool project @louisgv. I did a server implementation of the nonstreaming API, which didn't break the streaming version of the API and seems to work as expected for nonstreaming as well. Disclaimer: this is my first time writing anything beyond Hello World in Rust.. The implementation approach:
If you want to quickly test the implementation, here's a request body for the nonstreaming API:
If the implementation seems good enough on the server side, I could proceed and create support for it on the client side as well & make a PR. |
@JNeuvonen awesome :D - feel free to open a PR! (it makes reviewing it a bit easier for me :P) I will take a deeper look in a bit |
Support a non-stream version of the API. Without streaming, it's... more tricky (?). Since we need to load the model and do all kind of jazz before we can send them back a response.
Lack of streaming would likely requires:
Would need to experiment and see... but low priority because I personally don't use non-stream API :p... (any taker?)
The text was updated successfully, but these errors were encountered: