Fusional approaches for temporal action localization in untrimmed videos

This repo holds the codes and models for the framework, introduced in the paper:

"Hear Me Out: Fusional Approaches for AudioAugmented Temporal Action Localization".

• Paper Introduction
• Prerequisites
• Data setup
• Download datasets
• Training
• Testing
• Other info
  • citation
  • contact

State of the art architectures for untrimmed video Temporal Action Localization(TAL) have only considered RGB and Flow modalities, leaving the information-richaudio modality totally unexploited. Audio fusion has been explored for the relatedbut arguably easier problem of trimmed (clip-level) action recognition. However, TALposes a unique set of challenges. In this paper, we propose simple but effective fusion-based approaches for TAL. To the best of our knowledge, our work is the first to jointlyconsider audio and video modalities for supervised TAL. We experimentally show thatour schemes consistently improve performance for state of the art video-only TAL ap-proaches. Specifically, they help achieve new state of the art performance on large-scale benchmark datasets - ActivityNet-1.3 (52.73 [email protected]) and THUMOS14 ([email protected]). Our experiments include ablations involving multiple fusion schemes, modality combinations and TAL architectures.

The following table showcases the improvement in mAP scores due to incorporation of audio in current SOTA video-only architectures.

The training and testing in AVFusion is implemented in PyTorch for the ease of use.

• [PyTorch 1.8.1][pytorch]

Other minor Python modules can be installed by running

pip install -r requirements.txt

The code relies on CUDA extensions. Build them with the following command:

python setup.py develop

[back to top]

Clone this repo with git, please remember to use --recursive

git clone --recursive https://github.com/skelemoa/tal-hmo

[back to top]

[back to top]

Train your own models with the following command

python train_net.py  DATASET  --cfg CFG_PATH --snapshot_pref SNAPSHOT_PREF --epochs 20

SNAPSHOT_PREF: the path to save trained models and logs, e.g outputs/snapshpts/thumos14_rgb/.

We provide a script that finishes all steps on THUMOS14, including training, testing, and two-stream fusion. Run

bash scripts/do_all.sh

[back to top]

You can obtain the detection scores by running

sh test.sh TRAINING_CHECKPOINT

Here, TRAINING_CHECKPOINT denotes for the trained model. This script will report the detection performance in terms of [mean average precision][map] at different IoU thresholds.

The trained models and evaluation results are put in the "results" folder.

[back to top]

For questions and suggestions, file an issue or contact Jazib Mahmood at "[email protected]".

[back to top]