News
02/12/2023
We release ENeRF object-compositional representation code including training and visualization for ENeRF-Outdoor dataset.01/10/2023
We release ENeRF-Outdoor dataset.
Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
Haotong Lin*, Sida Peng*, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao and Xiaowei Zhou
SIGGRAPH Asia 2022 conference track
Project Page
conda create -n enerf python=3.8
conda activate enerf
pip install torch==1.9.0 cu111 torchvision==0.10.0 cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html # Important!
pip install -r requirements.txt
The workspace is the disk directory that stores datasets, training logs, checkpoints and results. Please ensure it has enough space.
export workspace=$PATH_TO_YOUR_WORKSPACE
Download the pretrained model from dtu_pretrain (Pretrained on DTU dataset.)
Put it into $workspace/trained_model/enerf/dtu_pretrain/latest.pth
.
Download the preprocessed DTU training data and Depth_raw from original MVSNet repo and unzip. MVSNeRF provide a DTU example, please follow with the example's folder structure.
mv dtu_example.zip $workspace
cd $workspace
unzip dtu_example.zip
This script only shows the example directory structure. You should download all the scenes in the DTU dataset and organize the data according to the example directory structure. Otherwise you can only do evaluation and fine-tuning on the example data.
Download the NeRF Synthetic and Real Forward-facing datasets from NeRF and unzip them to $workspace. You should have the following directory.
$workspace/nerf_llff_data
$workspace/nerf_synthetic
Download the ZJU-MoCap dataset from NeuralBody. Put it into $workspace/zju_mocap/CoreView_313.
Download the ENeRF-Outdoor dataset from this link. Put it into $workspace/enerf_outdoor/actor1.
Use the following command to train a generalizable model on DTU.
python train_net.py --cfg_file configs/enerf/dtu_pretrain.yaml
Our code also supports multi-gpu training. The published pretrained model was trained for 138000 iterations with 4 GPUs.
python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/enerf/dtu_pretrain.yaml distributed True gpus 0,1,2,3
cd $workspace/trained_model/enerf
mkdir dtu_ft_scan114
cp dtu_pretrain/138.pth dtu_ft_scan114
cd $codespace # codespace is the directory of the ENeRF code
python train_net.py --cfg_file configs/enerf/dtu/scan114.yaml
Fine-tuning for 3000 and 11000 iterations takes about 11 minutes and 40 minutes, respectively, on our test machine ( i9-12900K CPU, RTX 3090 GPU).
python train_net.py --cfg_file configs/enerf/zjumocap/zjumocap_train.yaml
python train_net.py --cfg_file configs/enerf/enerf_outdoor/actor1.yaml
Use the following command to evaluate the pretrained model on DTU.
python run.py --type evaluate --cfg_file configs/enerf/dtu_pretrain.yaml enerf.cas_config.render_if False,True enerf.cas_config.volume_planes 48,8 enerf.eval_depth True
{'psnr': 27.60513418439332, 'ssim': 0.9570619, 'lpips': 0.08897018397692591}
{'abs': 4.2624497, 'acc_2': 0.8003020328362158, 'acc_10': 0.9279663826227568}
{'mvs_abs': 4.4139433, 'mvs_acc_2': 0.7711405202036934, 'mvs_acc_10': 0.9262374398033109}
FPS: 21.778975517304048
21.8 FPS@512x640 is tested on a desktop with an Intel i9-12900K CPU and an RTX 3090 GPU. Add the "save_result True" parameter at the end of the command to save the rendering result.
python run.py --type evaluate --cfg_file configs/enerf/nerf_eval.yaml
python run.py --type evaluate --cfg_file configs/enerf/llff_eval.yaml
python run.py --type evaluate --cfg_file configs/enerf/zjumocap_eval.yaml
==============================
CoreView_313_level1 psnr: 31.48 ssim: 0.971 lpips:0.042
{'psnr': 31.477305846323087, 'ssim': 0.9714806, 'lpips': 0.04184799361974001}
==============================
FPS: 49.24468263992353
python run.py --type visualize --cfg_file configs/enerf/enerf_outdoor/actor1_path.yaml
We release the interactive rendering GUI for ZJU-MoCap dataset.
python gui_human.py --cfg_file configs/enerf/interactive/zjumocap.yaml
Usage:
Mouse wheel: Zoom in/out
Mouse left button: Move
Mouse right button: Rotate
Keyboard a: Align # Hold down a and then use the mouse right button to rotate the object for a good rendering trajectory
Keyboard s: Snap
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{lin2022enerf,
title={Efficient Neural Radiance Fields for Interactive Free-viewpoint Video},
author={Lin, Haotong and Peng, Sida and Xu, Zhen and Yan, Yunzhi and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
booktitle={SIGGRAPH Asia Conference Proceedings},
year={2022}
}