Panda WebUI is a lightweight and user-friendly web interface designed for running lightweight Language Models (LLMs) using the Hugging Face transformers library. In addition to supporting standard LLMs, Panda WebUI also accommodates GGUF models and several popular large Vision-Language Models (VLMs).
- Create a new environment
conda create -n panda python=3.11
conda activate panda- Install pytorch
pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121- For
llama-cpp(GGUF inference) anddeepspeed(training):
conda install -y -c "nvidia/label/cuda-12.1.1" cudaIf training is not needed, one can install the CUDA Runtime instead:
conda install -y -c "nvidia/label/cuda-12.1.1" cuda-runtime- Install the packages in requirements.txt
- Create a symbolic link
weightsto the model directory
Set the Huggingface cache by setting the environment variable in .bashrc:
export HF_HOME=/path/to/cache
This shell script can be used to run from the terminal.
cd ~/path/to/Panda-LLM/ || { echo "Failed to change directory"; exit 1; }
# Activate the conda environment named 'panda'
source ~/miniforge3/etc/profile.d/conda.sh
conda activate panda
# Check if the conda environment 'panda' is active.
if [ "$CONDA_DEFAULT_ENV" = "panda" ]; then
echo "Successfully activated conda environment 'panda'."
else
echo "Failed to activate conda environment 'panda'."
exit 1
fi
# Run the Python script
python webui.py