exo: Run your own AI cluster at home with everyday devices. Maintained by exo labs.

Discord | Telegram | X

Forget expensive NVIDIA GPUs, unify your existing devices into one powerful GPU: iPhone, iPad, Android, Mac, Linux, pretty much any device!

exo is experimental software. Expect bugs early on. Create issues so they can be fixed. The exo labs team will strive to resolve issues quickly.

We also welcome contributions from the community. We have a list of bounties in this sheet.

exo supports LLaMA and other popular models.

exo optimally splits up models based on the current network topology and device resources available. This enables you to run larger models than you would be able to on any single device.

exo will automatically discover other devices using the best method available. Zero manual configuration.

exo provides a ChatGPT-compatible API for running models. It's a one-line change in your application to run models on your own hardware using exo.

Unlike other distributed inference frameworks, exo does not use a master-worker architecture. Instead, exo devices connect p2p. As long as a device is connected somewhere in the network, it can be used to run models.

Exo supports different partitioning strategies to split up a model across devices. The default partitioning strategy is ring memory weighted partitioning. This runs an inference in a ring where each device runs a number of model layers proportional to the memory of the device.

The current recommended way to install exo is from source.

Python>=3.12.0 is required because of issues with asyncio in previous versions.

git clone https://github.com/exo-explore/exo.git
cd exo
pip install -r requirements.txt

python3 main.py

python3 main.py

That's it! No configuration required - exo will automatically discover the other device(s).

The native way to access models running on exo is using the exo library with peer handles. See how in this example for Llama 3.

exo also starts a ChatGPT-compatible API endpoint on http://localhost:8000. Note: this is currently only supported by tail nodes (i.e. nodes selected to be at the end of the ring topology). Example request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "llama-3-70b",
     "messages": [{"role": "user", "content": "What is the meaning of exo?"}],
     "temperature": 0.7
   }'

curl -X POST http://localhost:8001/api/v1/chat -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "What is the meaning of life?"}]}'

exo supports the following inference engines:

• ✅ MLX
• ✅ tinygrad
• 🚧 llama.cpp

• ✅ GRPC
• 🚧 Radio
• 🚧 Bluetooth

• 🚧 As the library is evolving so quickly, the iOS implementation has fallen behind Python. This is being worked on, and longer term we will push out an approach that will unify the implementations so we don't have to maintain separate implementations.