• [2024.08.02] 🔥 EchoMimic is now available on huggingface with A100 GPU. Thanks Wenmeng Zhou@ModelScope.
• [2024.07.25] 🔥🔥🔥 Accelerated models and pipe on Audio Driven are released. The inference speed can be improved by 10x (from ~7mins/240frames to ~50s/240frames on V100 GPU)
• [2024.07.23] 🔥 EchoMimic gradio demo on modelscope is ready.
• [2024.07.23] 🔥 EchoMimic gradio demo on huggingface is ready. Thanks Sylvain Filoni@fffiloni.
• [2024.07.17] 🔥🔥🔥 Accelerated models and pipe on Audio Selected Landmarks are released. The inference speed can be improved by 10x (from ~7mins/240frames to ~50s/240frames on V100 GPU)
• [2024.07.14] 🔥 ComfyUI is now available. Thanks @smthemex for the contribution.
• [2024.07.13] 🔥 Thanks NewGenAI for the video installation tutorial.
• [2024.07.13] 🔥 We release our pose&audio driven codes and models.
• [2024.07.12] 🔥 WebUI and GradioUI versions are released. We thank @greengerong @Robin021 and @O-O1024 for their contributions.
• [2024.07.12] 🔥 Our paper is in public on arxiv.
• [2024.07.09] 🔥 We release our audio driven codes and models.

（Some demo images above are sourced from image websites. If there is any infringement, we will immediately remove them and apologize.）

  git clone https://github.com/BadToBest/EchoMimic
  cd EchoMimic

• Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
• Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
• Tested Python Version: 3.8 / 3.10 / 3.11

Create conda environment (Recommended):

  conda create -n echomimic python=3.8
  conda activate echomimic

Install packages with pip

  pip install -r requirements.txt

Download and decompress ffmpeg-static, then

export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static

git lfs install
git clone https://huggingface.co/BadToBest/EchoMimic pretrained_weights

The pretrained_weights is organized as follows.

./pretrained_weights/
├── denoising_unet.pth
├── reference_unet.pth
├── motion_module.pth
├── face_locator.pth
├── sd-vae-ft-mse
│   └── ...
├── sd-image-variations-diffusers
│   └── ...
└── audio_processor
    └── whisper_tiny.pt

In which denoising_unet.pth / reference_unet.pth / motion_module.pth / face_locator.pth are the main checkpoints of EchoMimic. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:

• sd-vae-ft-mse
• sd-image-variations-diffusers
• audio_processor(whisper)

Run the python inference script:

  python -u infer_audio2vid.py
  python -u infer_audio2vid_pose.py

Edit the inference config file ./configs/prompts/animation.yaml, and add your own case:

test_cases:
  "path/to/your/image":
    - "path/to/your/audio"

The run the python inference script:

  python -u infer_audio2vid.py

(Firstly download the checkpoints with '_pose.pth' postfix from huggingface)

Edit driver_video and ref_image to your path in demo_motion_sync.py, then run

  python -u demo_motion_sync.py

Edit ./configs/prompts/animation_pose.yaml, then run

  python -u infer_audio2vid_pose.py

Set draw_mouse=True in line 135 of infer_audio2vid_pose.py. Edit ./configs/prompts/animation_pose.yaml, then run

  python -u infer_audio2vid_pose.py

Thanks to the contribution from @Robin021:

python -u webgui.py --server_port=3000

We would like to thank the contributors to the AnimateDiff, Moore-AnimateAnyone and MuseTalk repositories, for their open research and exploration.

We are also grateful to V-Express and hallo for their outstanding work in the area of diffusion-based talking heads.

If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.

If you find our work useful for your research, please consider citing the paper :

@misc{chen2024echomimic,
  title={EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning},
  author={Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma},
  year={2024},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}