https://github.com/mlc-ai/web-llm or something like it?

This is useful if you can only do a limited amount of processing on the server, and where the client has more processing power. WASM approaches to running Llama.cpp exist, as well as web GPU approaches (that only work on certain systems and and certain browser versions). There may be a WASM memory size issue with normal LLM sizes though. Working around it may well be possible though.

@justinh-rahb yep, seems promising!

@justinh-rahb yep, seems promising!

Yeah MLC-LLM is the ship!
#1270

Would love to plug-n-play with it