This repository contains the source code of AICSD (Adaptive Inter-Class Similarity Distillation for Semantic Segmentation ).

Intra-class distributions for each class. Distributions are created by applying softmax to spatial dimension of output prediction of last layer. Similarities between each pair of intra-class distributions have good potential for distillation. Distributions are created from the PASCAL VOC 2012 dataset with 21 category classes.

Overall diagram of the proposed AICSD. Network outputs are flattened into 1D vectors, followed by application of a softmax function to create intra-class distributions. KL divergence is then calculated between each distribution to create inter-class similarity matrices. An MSE loss function is then defined between the ICS matrices of the teacher and student. Also, KL divergence is calculated between the logits of the teacher and student for pixel-wise distillation. To mitigate the negative effects of teacher network, an adaptive weighting loss strategy is used to scale two distillation losses and ross-entropy loss of semantic segmentation. During training, hyperparameter $\alpha$ undergoes adaptive changes and progressively increases with epoch number.

For ICSD method:

python train_kd.py --backbone resnet18 --dataset pascal  --pa_lambda 9500

For AICSD method:

python train_kd.py --backbone resnet18 --dataset pascal  --pa_lambda 9500 --pi_lambda 10 --ALW

Download following pre-trained teacher network and put it into pretrained/ directory

• ResNet101-DeepLabV3

measure performance on test set with Pascal VOC evaluation server.

If you use this repository for your research or wish to refer to our distillation method, please use the following BibTeX entry:

@article{mansourian2023aicsd,
  title={AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation},
  author={Mansourian, Amir M and Ahmadi, Rozhan and Kasaei, Shohreh},
  journal={arXiv preprint arXiv:2308.04243},
  year={2023}
}

This codebase is heavily borrowed from A Comprehensive Overhaul of Feature Distillation . Thanks for their excellent work.