Code for the paper: All in One: Exploring Unified Video-Language Pre-training Arxiv
- 2022.03.25 Update Readme.
- 2022.06.07 Release the model AllInOne pre-trained on Eight Dataset (YTT WebVid HowTo CC3 CC12 CoCo VG SBU).
- 2022.05.07 AllInOne is released. The main different between AllInOne is the Image and Video Co-train.
- 2022.03.14 The first version of AllInOne is released.
In this work, we use PytorchLighting for distributed training with mixed precision. Install pytorch and PytorchLighting first.
conda create -n allinone python=3.7
source activate allinone
cd [Path_To_This_Code]
pip install -r requirements.txtIf all packages include ffmpeg installed, please skip step 2.
To speed up the pre-training, we adopt on-the-fly decode for fast IO. Install ffmpeg as below.
sudo conda install -y ffmpegPlease install the required packages if not included in the requirements.txt.
If you server cannot connect to http or install ffmpeg slowly. Please download static binary file from FFmpeg Static Builds and then add to path variable, as follows:
export PATH=[PATH_TO_Dir/]ffmpeg-git-20220108-amd64-static:$PATHInstall pytorchvideo (for data augmentation) as below:
pip install ffmpeg-python
pip install pytorchvideoWe provide three pretrained weights in google driver.
| Model | PT Data | Parameter | Pretrained Weight | Trained Log | Hparams |
|---|---|---|---|---|---|
| All-in-one-Ti | Webvid HowTo | 12M | Google Driver | Google Driver | Google Driver |
| All-in-one-S | Webvid HowTo | 33M | Google Driver | Google Driver | Google Driver |
| All-in-one-B | Webvid HowTo | 110M | Google Driver | Google Driver | Google Driver |
| All-in-one-B | Webvid HowTo CC3 |
110M | Google Driver | Google Driver | Google Driver |
| All-in-one-B | Webvid YTT HowTo CC3 CC12 Coco VG SBU |
110M | Google Driver | Google Driver | Google Driver |
After downloaded these pretrained weights, move them into pretrained dir.
mkdir pretrained
cp *.ckpt pretrained/| Model | Param | Data | Frames | TGIF-Action | TGIF-Frame | MSR R@5 | MSR R@10 |
|---|---|---|---|---|---|---|---|
| ClipBERT | 137M | I:Coco VG | 8 x 2 | 82.9 | 59.4 | 49.2 | 63.5 |
| VIOLET | 198M | V:Webvid I:CC3 |
16 | 87.1 | - | 63.0 | 73.4 |
| All-in-one-S | 33M | V:WebVid Howto | 3 | 91.2 | 64.0 | 61.5 | 70.9 |
| All-in-one-B | 110M | V:WebVid Howto | 3 | 92.9 | 64.2 | 67.0 | 77.1 |
| All-in-one-B | 110M | V:Webvid I:CC3 |
3 | 95.4 | 67.2 | 68.1 | 77.3 |
| All-in-one-B | 110M | V:Webvid YTT HowTo I:CC3 CC12 Coco VG SBU |
3 | 96.3 | 68.5 | 70.3 | 79.2 |
I is short for Image and V is short for Video in this table.
See DATA.md
See TRAIN.md
See COTRAIN.md
See EVAL.md
By unified design and sparse sampling, AllInOne show much small flops.
If you find our work helps, please cite our paper.
@article{wang2022allinone,
title={All in One: Exploring Unified Video-Language Pre-training},
author={Wang, Alex Jinpeng and Ge, Yixiao and Yan, Rui and Ge Yuying and Lin, Xudong and Cai, Guanyu and Wu, Jianping and Shan, Ying and Qie, Xiaohu and Shou, Mike Zheng},
journal={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2023}
}Email: awinyimgprocess at gmail dot com
If you have any problem or have difficult in reproducing the results reported in this code, you can email to me or open a question in issues. We are also willing to merge the code if transfer our All-in-one to different tasks or datasets.
This work is mainly based on ViLT, Frozen and Merlot.
MIT