In colab change this file /usr/local/lib/python3.10/dist-packages/diffusers/models/init.py to https://github.com/G-force78/FRESCO/blob/main/_init_.py

FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
Shuai Yang, Yifan Zhou, Ziwei Liu and Chen Change Loy
in CVPR 2024
Project Page | Paper | Supplementary Video | Input Data and Video Results

Abstract: The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.

Features:

• Temporal consistency: use intra-and inter-frame constraint with better consistency and coverage than optical flow alone.
  • Compared with our previous work Rerender-A-Video, FRESCO is more robust to large and quick motion.
• Zero-shot: no training or fine-tuning required.
• Flexibility: compatible with off-the-shelf models (e.g., ControlNet, LoRA) for customized translation.

• [03/2023] Paper is released.
• [03/2023] Code is released.
• [03/2024] This website is created.

• Integrate into Diffusers
• Add Huggingface web demo
• Add webUI.
• Update readme
• Upload paper to arXiv, release related material

1. Clone the repository.

git clone https://github.com/williamyang1991/FRESCO.git
cd FRESCO

2. You can simply set up the environment with pip based on requirements.txt

   • We have tested on torch 2.0.0/2.1.0 and diffusers 0.19.3
   • If you use new versions of diffusers, you need to modify my_forward()

3. Run the installation script. The required models will be downloaded in ./model, ./src/ControlNet/annotator and ./src/ebsynth/deps/ebsynth/bin.

   • Requires access to huggingface.co

python install.py

4. You can run the demo with run_fresco.py

python run_fresco.py ./config/config_music.yaml

5. For issues with Ebsynth, please refer to issues

python webUI.py

The Gradio app also allows you to flexibly change the inference options. Just try it for more details.

Upload your video, input the prompt, select the model and seed, and hit:

• Run Key Frames: detect keyframes, translate all keyframes.
• Run Propagation: propagate the keyframes to other frames for full video translation
• Run All: Run Key Frames and Run Propagation

Select the model:

• Base model: base Stable Diffusion model (SD 1.5)
  • Stable Diffusion 1.5: official model
  • rev-Animated: a semi-realistic (2.5D) model
  • realistic-Vision: a photo-realistic model
  • flat2d-animerge: a cartoon model
  • You can add other models on huggingface.co by modifying this line

We provide abundant advanced options to play with

We provide a flexible script run_fresco.py to run our method.

Set the options via a config file. For example,

python run_fresco.py ./config/config_music.yaml

We provide some examples of the config in config directory. Most options in the config is the same as those in WebUI. Please check the explanations in the WebUI section.

We provide a separate Ebsynth python script video_blend.py with the temporal blending algorithm introduced in Stylizing Video by Example for interpolating style between key frames. It can work on your own stylized key frames independently of our FRESCO algorithm. For the details, please refer to our previous work Rerender-A-Video

a red car turns in the winter	an African American boxer wearing black boxing gloves punches towards the camera, cartoon style	a cartoon spiderman in black suit, black shoes and white gloves is dancing	a beautiful woman holding her glasses in CG style

If you find this work useful for your research, please consider citing our paper:

@inproceedings{yang2024fresco,
 title = {FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation},
 author = {Yang, Shuai and Zhou, Yifan and Liu, Ziwei and and Loy, Chen Change},
 booktitle = {CVPR},
 year = {2024},
}

The code is mainly developed based on Rerender-A-Video, ControlNet, Stable Diffusion, GMFlow and Ebsynth.