Code for Learning from Synthetic Aniamls (CVPR 2020, oral). The code is developed based on the Pytorch framework(1.1.0) with python 3.7.3. This repo includes training code for consistency-constrained semi-supervised learning plus a synthetic animal dataset.

If you find our code or method helpful, please use the following BibTex entry.

@InProceedings{Mu_2020_CVPR,
author = {Mu, Jiteng and Qiu, Weichao and Hager, Gregory D. and Yuille, Alan L.},
title = {Learning From Synthetic Animals},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

• PyTorch (tested with 1.1.0): Please follow the installation instruction of PyTorch.
• Other packages, e.g. imgaug 0.3.0, opencv 4.1.1, scipy 1.2.1

1. Clone the repository with submodule.

   git clone https://github.com/JitengMu/Learning-from-Synthetic-Animals
   

2. Go to directory Learning-from-Synthetic-Animals/ and create a symbolic link to the images directory of the animal dataset:

   ln -s PATH_TO_IMAGES_DIR ./animal_data
   

3. Download and pre-processing datasets:

   • Download TigDog Dataset and move folder behaviorDiscovery2.0 to ./animal_data.
   • Run python get_cropped_TigDog.py to get cropped TigDog Dataset.
   • Download Synthetic Animal Dataset with script bash get_dataset.sh.

• ./data for train/val split and dataset statistics (mean/std).

• ./train for training scripts.

• ./evaluation for inference scripts

• ./CC-SSL for consistency constrained semi-supervised learing.

• ./pose for stacked hourglass model.

• ./data_generation for synthetic animal dataset generation.

1. Download the checkpoint with script bash get_checkpoint.sh and the structure looks like:

checkpoint    
│
└───real_animal
│      │   horse
│      └   tiger
│    
└───synthetic_animal
       │   horse
       │   tiger
       └   others
   

2. Run the demo.ipynb to visualize predictions.

3. Evaluate the accuracy on TigDog Dataset. (18 per joint accuracies are in the order of left-eye, right-eye, chin, left-front-hoof, right-front-hoof, left-back-hoof, right-back-hoof, left-front-knee, right-front-knee, left-back-knee, right-back-knee, left-shoulder, right-shoulder, left-front-elbow, right-front-elbow, left-back-elbow, right-back-elbow)

CUDA_VISIBLE_DEVICES=0 python ./evaluation/test.py --dataset1 synthetic_animal_sp --dataset2 real_animal_sp --arch hg --resume ./checkpoint/synthetic_animal/horse/horse_ccssl/synthetic_animal_sp.pth.tar --evaluate --animal horse

1. Training on synthetic animal dataset.

CUDA_VISIBLE_DEVICES=0 python train/train.py --dataset synthetic_animal_sp -a hg --stacks 4 --blocks 1 --image-path ./animal_data/ --checkpoint ./checkpoint/horse/syn --animal horse

2. Generate pseudo-labels for TigDog dataset and jointly train on synthetic animal and TigDog datasets.

CUDA_VISIBLE_DEVICES=0 python CCSSL/CCSSL.py --num-epochs 60 --checkpoint ./checkpoint/horse/ssl/ --resume ./checkpoint/horse/syn/model_best.pth.tar --animal horse

3. Evaluate the accuracy on TigDog Dataset using metric [email protected].

CUDA_VISIBLE_DEVICES=0 python ./evaluation/test.py --dataset1 synthetic_animal_sp --dataset2 real_animal_sp --arch hg --resume ./checkpoint/horse/ssl/synthetic_animal_sp.pth.tar --evaluate --animal horse

Please refer to TRAINING.md for detailed training recipes!

1. Download and install the unrealcv_binary for Linux (tested in Ubuntu 16.04) with bash get_unrealcv_binary.sh

2. Run unreal engine.

DISPLAY= ./data_generation/unrealcv_binary/LinuxNoEditor/AnimalParsing/Binaries/Linux/AnimalParsing -cvport 9900

3. Run the following script to generate images and ground truth (images, depths, keypoints)

python data_generation/unrealdb/example/animal_example/animal_data.py --animal horse --random-texture-path ./data_generation/val2017/ --use-random-texture --num-imgs 10

Generated data is saved in ./data_generation/generated_data/ by default.

4. Run the following script to generate part segmentations (support horse, tiger)

python data_generation/generate_partseg.py --animal horse --dataset-path ./data_generation/generated_data/

• Wei Yang's Stacked Hourglass Model

• Yunsheng Li's Bidirectional Learning for Domain Adaptation of Semantic Segmentation