流式推理，合入 fast_inference 分支中 flash attention 等推理加速相关变更

注意: WebUI的流式推理需要 gradio==4.27.0，请先 runtime\python.exe -m pip install -U gradio==4.27.0

1. WebUI：runtime\python.exe GPT_SoVITS/inference_stream_webui.py (因为Windows浏览器策略限制，需要点击播放按钮才会播放；MacOS可以修改 audio_file = gr.Audio(autoplay=True)在推理后自动播放音频)
2. API：uvicorn GPT_SoVITS.inference_stream_api:app --host 0.0.0.0 --port 5000

• 然后访问浏览器：http://localhost:5000/tts?text=这里是需要推理的文本 (其他参数见inference_stream_api.py)

WebUI流式推理演示 (Windows)

stream_webui.mp4

API模式流式推理

win_streaming.mp4

1. Zero-shot TTS: Input a 5-second vocal sample and experience instant text-to-speech conversion.

2. Few-shot TTS: Fine-tune the model with just 1 minute of training data for improved voice similarity and realism.

3. Cross-lingual Support: Inference in languages different from the training dataset, currently supporting English, Japanese, and Chinese.

4. WebUI Tools: Integrated tools include voice accompaniment separation, automatic training set segmentation, Chinese ASR, and text labeling, assisting beginners in creating training datasets and GPT/SoVITS models.

Check out our demo video here!

Unseen speakers few-shot fine-tuning demo:

User guide: 简体中文 | English

For users in China region, you can click here to use AutoDL Cloud Docker to experience the full functionality online.

• Python 3.9, PyTorch 2.0.1, CUDA 11
• Python 3.10.13, PyTorch 2.1.2, CUDA 12.3
• Python 3.9, PyTorch 2.2.2, macOS 14.4.1 (Apple silicon)
• Python 3.9, PyTorch 2.2.2, CPU devices

Note: numba==0.56.4 requires py<3.11

If you are a Windows user (tested with win>=10), you can directly download the pre-packaged distribution and double-click on go-webui.bat to start GPT-SoVITS-WebUI.

Users in China region can download the 0217 package or the 0306fix2 package by clicking the links and then selecting "Download a copy."

Note: The 0306fix2 version doubles the inference speed and fixes all issues with the no reference text mode.

conda create -n GPTSoVits python=3.9
conda activate GPTSoVits
bash install.sh

Note: The models trained with GPUs on Macs result in significantly lower quality compared to those trained on other devices, so we are temporarily using CPUs instead.

1. Install Xcode command-line tools by running xcode-select --install
2. Install FFmpeg by running brew install ffmpeg or conda install ffmpeg.
3. Install the program by running the following commands:

conda create -n GPTSoVits python=3.9
conda activate GPTSoVits

pip install -r requirements.txt

pip install -r requirements.txt

conda install ffmpeg

sudo apt install ffmpeg
sudo apt install libsox-dev
conda install -c conda-forge 'ffmpeg<7'

Download and place ffmpeg.exe and ffprobe.exe in the GPT-SoVITS root.

0. Regarding image tags: Due to rapid updates in the codebase and the slow process of packaging and testing images, please check Docker Hub for the currently packaged latest images and select as per your situation, or alternatively, build locally using a Dockerfile according to your own needs.
1. Environment Variables：

• is_half: Controls half-precision/double-precision. This is typically the cause if the content under the directories 4-cnhubert/5-wav32k is not generated correctly during the "SSL extracting" step. Adjust to True or False based on your actual situation.

2. Volumes Configuration，The application's root directory inside the container is set to /workspace. The default docker-compose.yaml lists some practical examples for uploading/downloading content.
3. shm_size： The default available memory for Docker Desktop on Windows is too small, which can cause abnormal operations. Adjust according to your own situation.
4. Under the deploy section, GPU-related settings should be adjusted cautiously according to your system and actual circumstances.

docker compose -f "docker-compose.yaml" up -d

As above, modify the corresponding parameters based on your actual situation, then run the following command:

docker run --rm -it --gpus=all --env=is_half=False --volume=G:\GPT-SoVITS-DockerTest\output:/workspace/output --volume=G:\GPT-SoVITS-DockerTest\logs:/workspace/logs --volume=G:\GPT-SoVITS-DockerTest\SoVITS_weights:/workspace/SoVITS_weights --workdir=/workspace -p 9880:9880 -p 9871:9871 -p 9872:9872 -p 9873:9873 -p 9874:9874 --shm-size="16G" -d breakstring/gpt-sovits:xxxxx

Download pretrained models from GPT-SoVITS Models and place them in GPT_SoVITS/pretrained_models.

For UVR5 (Vocals/Accompaniment Separation & Reverberation Removal, additionally), download models from UVR5 Weights and place them in tools/uvr5/uvr5_weights.

Users in China region can download these two models by entering the links below and clicking "Download a copy"

• GPT-SoVITS Models

• UVR5 Weights

For Chinese ASR (additionally), download models from Damo ASR Model, Damo VAD Model, and Damo Punc Model and place them in tools/asr/models.

For English or Japanese ASR (additionally), download models from Faster Whisper Large V3 and place them in tools/asr/models. Also, other models may have the similar effect with smaller disk footprint.

Users in China region can download this model by entering the links below

• Faster Whisper Large V3 (clicking "Download a copy")

• Faster Whisper Large V3 (HuggingFace mirror site)

The TTS annotation .list file format:

vocal_path|speaker_name|language|text

Language dictionary:

• 'zh': Chinese
• 'ja': Japanese
• 'en': English

Example:

D:\GPT-SoVITS\xxx/xxx.wav|xxx|en|I like playing Genshin.

• High Priority:

  • Localization in Japanese and English.
  • User guide.
  • Japanese and English dataset fine tune training.

• Features:

  • Zero-shot voice conversion (5s) / few-shot voice conversion (1min).
  • TTS speaking speed control.
  • Enhanced TTS emotion control.
  • Experiment with changing SoVITS token inputs to probability distribution of vocabs.
  • Improve English and Japanese text frontend.
  • Develop tiny and larger-sized TTS models.
  • Colab scripts.
  • Try expand training dataset (2k hours -> 10k hours).
  • better sovits base model (enhanced audio quality)
  • model mix

Use the command line to open the WebUI for UVR5

python tools/uvr5/webui.py "<infer_device>" <is_half> <webui_port_uvr5>

If you can't open a browser, follow the format below for UVR processing,This is using mdxnet for audio processing

python mdxnet.py --model --input_root --output_vocal --output_ins --agg_level --format --device --is_half_precision 

This is how the audio segmentation of the dataset is done using the command line

python audio_slicer.py \
    --input_path "<path_to_original_audio_file_or_directory>" \
    --output_root "<directory_where_subdivided_audio_clips_will_be_saved>" \
    --threshold <volume_threshold> \
    --min_length <minimum_duration_of_each_subclip> \
    --min_interval <shortest_time_gap_between_adjacent_subclips> 
    --hop_size <step_size_for_computing_volume_curve>

This is how dataset ASR processing is done using the command line(Only Chinese)

python tools/asr/funasr_asr.py -i <input> -o <output>

ASR processing is performed through Faster_Whisper(ASR marking except Chinese)

(No progress bars, GPU performance may cause time delays)

python ./tools/asr/fasterwhisper_asr.py -i <input> -o <output> -l <language>

A custom list save path is enabled

Special thanks to the following projects and contributors:

• ar-vits
• SoundStorm
• vits
• TransferTTS
• contentvec
• hifi-gan
• fish-speech

• Chinese Speech Pretrain
• Chinese-Roberta-WWM-Ext-Large

• paddlespeech zh_normalization
• LangSegment

• ultimatevocalremovergui
• audio-slicer
• SubFix
• FFmpeg
• gradio
• faster-whisper
• FunASR