The official implementation of CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers (IEEE T-ITS 2023): More details can be found in our paper [PDF].

1. Requirements

• Python 3.7
• PyTorch 1.7.0 or higher
• CUDA 10.2 or higher

We have tested the following versions of OS and softwares:

• OS: Ubuntu 18.04.6 LTS
• CUDA: 10.2
• PyTorch 1.8.2
• Python 3.8.11

2. Install all dependencies. Install pytorch, cuda and cudnn, then install other dependencies via:

pip install -r requirements.txt

Orgnize the dataset folder in the following structure:

<datasets>
|-- <DatasetName1>
    |-- <RGBFolder>
        |-- <name1>.<ImageFormat>
        |-- <name2>.<ImageFormat>
        ...
    |-- <ModalXFolder>
        |-- <name1>.<ModalXFormat>
        |-- <name2>.<ModalXFormat>
        ...
    |-- <LabelFolder>
        |-- <name1>.<LabelFormat>
        |-- <name2>.<LabelFormat>
        ...
    |-- train.txt
    |-- test.txt
|-- <DatasetName2>
|-- ...

train.txt contains the names of items in training set, e.g.:

<name1>
<name2>
...

For RGB-Depth semantic segmentation, the generation of HHA maps from Depth maps can refer to https://github.com/charlesCXK/Depth2HHA-python.

For preparation of other datasets, please refer to the original websites:

• NYU Depth V2
• SUN-RGBD
• ScanNetV2
• RGB-T MFNet
• RGB-P ZJU
• RGB-E EventScape
• RGB-L KITTI-360

1. Pretrain weights:

   Download the pretrained segformer here pretrained segformer.

2. Config

   Edit config file in configs.py, including dataset and network settings.

3. Run multi GPU distributed training:

   $ CUDA_VISIBLE_DEVICES="GPU IDs" python -m torch.distributed.launch --nproc_per_node="GPU numbers you want to use" train.py

• The tensorboard file is saved in log_<datasetName>_<backboneSize>/tb/ directory.
• Checkpoints are stored in log_<datasetName>_<backboneSize>/checkpoints/ directory.

Run the evaluation by:

CUDA_VISIBLE_DEVICES="GPU IDs" python eval.py -d="Device ID" -e="epoch number or range"

If you want to use multi GPUs please specify multiple Device IDs (0,1,2...).

We offer the pre-trained weights on different RGBX datasets (Some weights are not available yet. Due to the difference of training platforms, these weights may not be correctly loaded):

If you find this repo useful, please consider referencing the following paper:

@article{zhang2023cmx,
  title={CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers},
  author={Zhang, Jiaming and Liu, Huayao and Yang, Kailun and Hu, Xinxin and Liu, Ruiping and Stiefelhagen, Rainer},
  journal={IEEE Transactions on Intelligent Transportation Systems},
  year={2023}
}

Our code is heavily based on TorchSeg and SA-Gate, thanks for their excellent work!