This repository contains the official source code for our paper:
ReSTR: Convolution-Free Referring Image Segmentation Using Transformers
Namyup Kim1, Dongwon Kim1, Cuiling Lan2, Wenjun Zeng2, and Suha Kwak1
1POSTECH CSE, 2Microsoft Research Asia
CVPR, 2022.
- Python 3.7.13
- PyTorch 1.13.1 cu117
Instructions:
conda env create -f restr.yaml
conda activate restr- Download or use symlink, such that the MS COCO images are under
data/coco/images/train2014/ - Download or use symlink, such that the ReferItGame data are under
data/referit/imagesanddata/referit/mask
- Download, git clone, or use symlink, such that refer is under
external. Then strictly follow theSetupandDownloadsection of its README. Also, put thereferfolder in PYTHONPATH asexport PYTHONPATH=${PYTHONPATH}:/my/restr/path/external/refer - Download, git clone, or use symlink, such that the MS COCO API is under
external(i.e.external/coco/PythonAPI/pycocotools)
python build_batches.py -d Gref -t train --img-size 480
python build_batches.py -d Gref -t val --img-size 480
python build_batches.py -d unc -t train --img-size 480
python build_batches.py -d unc -t val --img-size 480
python build_batches.py -d unc -t testA --img-size 480
python build_batches.py -d unc -t testB --img-size 480
python build_batches.py -d unc -t train --img-size 480
python build_batches.py -d unc -t val --img-size 480
python build_batches.py -d unc -t testA --img-size 480
python build_batches.py -d unc -t testB --img-size 480
python build_batches.py -d referit -t trainval --img-size 480
python build_batches.py -d referit -t test --img-size 480
├─ ./data
├─ mscoco
│ ├─ Gref_480_batch
│ │ ├─ train_batch
│ │ | ├─ Gref_train_0.npz
│ │ | ├─ Gref_train_1.npz
│ │ | └─ ...
| | ├─ train_image
│ │ ├─ train_label
│ │ ├─ val_batch
│ │ ├─ val_image
│ │ └─ val_label
│ ├─ unc_480_batch
│ └─ unc _480_batch
├─ referit
│ └─ referit_480_batch
│ ├─ trainval_batch
│ └─ text_batch
├─ Gref_emb.npy
├─ referit_emb.npy
├─ vocabulary_Gref.txt
└─ vocabulary_referit.txt
python train_restr.py --data_dir ./data/mscoco/Gref_480_batch --adamW
python train_restr.py --data_dir ./data/mscoco/unc_480_batch --adamW
python train_restr.py --data_dir ./data/mscoco/unc _480_batch --adamW
python train_restr.py --data_dir ./data/referit/referit_480_batch --set trainval --valset test --adamW
cd eval
python evaluate.py --data_dir ../data/mscoco/Gref_batch --restore_refseg ../weights/to/ --set val --iters 400000 --is_vis
@inproceedings{kim2022restr,
title={Restr: Convolution-free referring image segmentation using transformers},
author={Kim, Namyup and Kim, Dongwon and Lan, Cuiling and Zeng, Wenjun and Kwak, Suha},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={18145--18154},
year={2022}
}
This code is built upon the following public repositories.