A Gradio web UI for Large Language Models.

Its goal is to become the AUTOMATIC1111/stable-diffusion-webui of text generation.

• 3 interface modes: default (two columns), notebook, and chat
• Multiple model backends: transformers, llama.cpp, ExLlama, AutoGPTQ, GPTQ-for-LLaMa, ctransformers
• Dropdown menu for quickly switching between different models
• LoRA: load and unload LoRAs on the fly, train a new LoRA using QLoRA
• Precise instruction templates for chat mode, including Llama-2-chat, Alpaca, Vicuna, WizardLM, StableLM, and many others
• 4-bit, 8-bit, and CPU inference through the transformers library
• Use llama.cpp models with transformers samplers (llamacpp_HF loader)
• Multimodal pipelines, including LLaVA and MiniGPT-4
• Extensions framework
• Custom chat characters
• Very efficient text streaming
• Markdown output with LaTeX rendering, to use for instance with GALACTICA
• API, including endpoints for websocket streaming (see the examples)

To learn how to use the various features, check out the Documentation: https://github.com/oobabooga/text-generation-webui/tree/main/docs

Just download the zip above, extract it, and double-click on "start". The web UI and all its dependencies will be installed in the same folder.

• The source codes and more information can be found here: https://github.com/oobabooga/one-click-installers
• There is no need to run the installers as admin.
• Huge thanks to @jllllll, @ClayShoaf, and @xNul for their contributions to these installers.

Recommended if you have some experience with the command-line.

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands (source):

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

conda create -n textgen python=3.10.9
conda activate textgen

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

• MacOS users: oobabooga#393
• AMD users: https://rentry.org/eq3hg

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt

Precompiled wheels are included for CPU-only and NVIDIA GPUs (cuBLAS). For AMD, Metal, and some specific CPUs, you need to uninstall those wheels and compile llama-cpp-python yourself.

To uninstall:

pip uninstall -y llama-cpp-python llama-cpp-python-cuda

To compile: https://github.com/abetlen/llama-cpp-python#installation-with-openblas--cublas--clblast--metal

bitsandbytes >= 0.39 may not work. In that case, to use --load-in-8bit, you may have to downgrade like this:

• Linux: pip install bitsandbytes==0.38.1
• Windows: pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

ln -s docker/{Dockerfile,docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
# Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model
docker compose up --build

• You need to have docker compose v2.17 or higher installed. See this guide for instructions.
• For additional docker files, check out this repository.

From time to time, the requirements.txt changes. To update, use these commands:

conda activate textgen
cd text-generation-webui
pip install -r requirements.txt --upgrade

Models should be placed in the text-generation-webui/models folder. They are usually downloaded from Hugging Face.

• Transformers or GPTQ models are made of several files and must be placed in a subfolder. Example:

text-generation-webui
├── models
│   ├── lmsys_vicuna-33b-v1.3
│   │   ├── config.json
│   │   ├── generation_config.json
│   │   ├── pytorch_model-00001-of-00007.bin
│   │   ├── pytorch_model-00002-of-00007.bin
│   │   ├── pytorch_model-00003-of-00007.bin
│   │   ├── pytorch_model-00004-of-00007.bin
│   │   ├── pytorch_model-00005-of-00007.bin
│   │   ├── pytorch_model-00006-of-00007.bin
│   │   ├── pytorch_model-00007-of-00007.bin
│   │   ├── pytorch_model.bin.index.json
│   │   ├── special_tokens_map.json
│   │   ├── tokenizer_config.json
│   │   └── tokenizer.model

In the "Model" tab of the UI, those models can be automatically downloaded from Hugging Face. You can also download them via the command-line with python download-model.py organization/model.

• GGML/GGUF models are a single file and should be placed directly into models. Example:

text-generation-webui
├── models
│   ├── llama-13b.ggmlv3.q4_K_M.bin

Those models must be downloaded manually, as they are not currently supported by the automated downloader.

python download-model.py EleutherAI/gpt-j-6B --text-only

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

http://localhost:7860/?__theme=dark

Optionally, you can use the following command-line flags:

⚠️ Requires minimum compute of 7.0 on Windows at the moment.

Inference settings presets can be created under presets/ as yaml files. These files are detected automatically at startup.

The presets that are included by default are the result of a contest that received 7215 votes. More details can be found here.

If you would like to contribute to the project, check out the Contributing guidelines.

• Subreddit: https://www.reddit.com/r/oobabooga/
• Discord: https://discord.gg/jwZCF2dPQN

In August 2023, Andreessen Horowitz (a16z) provided a generous grant to encourage and support my independent work on this project. I am extremely grateful for their trust and recognition, which will allow me to dedicate more time towards realizing the full potential of text-generation-webui.