This repo was developed as part of our CVPRW '23 paper Parcel3D: Shape Reconstruction From Single RGB Images for Applications in Transportation Logistics (citation), also check the project page for more details.
Figure:
We take an RGB image as input and use Cube R-CNN's Cube Head to estimate a 3D bounding box. This bounding box is subdivided and serves as initial mesh, which is refined by an iterative mesh refinement. For training and evaluation we present Parcel3D, a novel dataset of normal and damaged parcels with full 3D annotations.
(c) IEEE 2023.
We highly recommend to use the provided Devcontainer to make the usage as easy as possible:
- Install Docker and VS Code
- Install VS Code Devcontainer extension
ms-vscode-remote.remote-containers
- Clone the repository
git clone https://github.com/a-nau/cuberefine-r-cnn.git
- Press
F1
(orCTRL SHIFT P
) and selectDev Containers: Rebuild and Reopen Container
- Go to
Run and Debug (CTRL SHIFT D)
and press the run button, alternatively pressF5
Alternatively, make sure to use CUDA 11.3 (see this repo if you use multiple CUDA versions) and Torch 1.11 and install the following
# General
pip install scikit-image==0.18.* matplotlib imageio==2.31.1 opencv-python==4.8.0.74 scipy==1.10.1 pandas==2.0.3 joblib==1.3.1 einops==0.6.1
# Detectron2
pip install "git https://github.com/facebookresearch/[email protected]"
# Pytorch3D
pip install fvcore==0.1.5.post20221221
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1110/download.html
# PyTorch Geometric
pip install pyg_lib==0.2.0 torch_scatter==2.0.9 torch_sparse==0.6.13 torch_cluster==1.6.0 torch_spline_conv==1.2.1 -f https://data.pyg.org/whl/torch-1.11.0 cu113.html
pip install torch_geometric==2.3.1
and run
python tools/train_parcel3d.py --config-file configs/Base_Parcel3D.yaml --num-gpus 1
You can download the Parcel3D dataset here.
Figure:
Visual samples from Parcel3D.
Check our project website for more.
Note that you need to use the json
files from the misc/omni3d_json
folder
If you use this code for scientific research, please consider citing the following works.
Our work for which this repository was developed.
@inproceedings{naumannParcel3DShapeReconstruction2023,
author = {Naumann, Alexander and Hertlein, Felix and D\"orr, Laura and Furmans, Kai},
title = {Parcel3D: Shape Reconstruction From Single RGB Images for Applications in Transportation Logistics},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2023},
pages = {4402-4412}
address = {Vancouver, Canada},
}
The work of Brazil et al. from which this code is heavily borrowed
@inproceedings{brazil2023omni3d,
title = {{Omni3D}: A Large Benchmark and Model for {3D} Object Detection in the Wild},
author = {Garrick Brazil and Abhinav Kumar and Julian Straub and Nikhila Ravi and Justin Johnson and Georgia Gkioxari},
year = 2023,
month = {June},
booktitle = {CVPR},
address = {Vancouver, Canada},
organization = {IEEE}
}
The work of Gkioxari et al. from which some code is borrowed
@inproceedings{Gkioxari_2019_ICCV,
title = {Mesh R-CNN},
author = {Gkioxari, Georgia and Malik, Jitendra and Johnson, Justin},
year = 2019,
month = {October},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}
}
This is a modified version of Cube R-CNN, which is released under CC-BY-NC 4.0. It also borrows part of the code from Mesh R-CNN, which is released under BSD-3-Clause License.
Unless otherwise stated, this repo is distributed under CC-BY-NC 4.0.