-
Atis (atis)
Preprocessed dataset stored in
data/atis. -
MultiWoZ (multiwoz)
Preprocessed dataset stored in
data/multiwoz. -
CLINC (clinc)
Preprocessed dataset stored in
data/clinc. -
Banking77 (bank)
Preprocessed dataset stored in
data/bank.
Each dataset stored in all.csv w/o splitting.
All predefined splits stored in dataset root directory.
All information about intents stored in intent_info folder.
intent_info/descriptions contains intent descriptions and different types of patterns.
Intent and utterance similarity matrices for negative sampling stored in intent_info/intent_similarity and uttr_similarity directories respectively.
data
| atis
| |-- all.csv
| | intent_info
| | |-- actions.json
| | |-- concepts.json
| | | descriptions
| | | |-- d1_pattern.json
| | | |-- names.json
| | | intent_similarity
| | | | simcse
| | | | |-- raws.json
| | | | |-- similarity.txt
| | uttr_similarity
| | |-- simcse_100.txt
| bank
| |-- all.csv
| | intent_info
| | |-- actions.json
| | |-- concepts.json
| | | descriptions
| | | |-- d1_pattern.json
| | | |-- names.json
| | | intent_similarity
| | | | simcse
| | | | |-- raws.json
| | | | |-- similarity.txt
| | uttr_similarity
| | |-- simcse_100.txt
| clinc
| |-- all.csv
| | intent_info
| | |-- actions.json
| | |-- concepts.json
| | | descriptions
| | | |-- d1_pattern.json
| | | |-- names.json
| | | intent_similarity
| | | | simcse
| | | | |-- raws.json
| | | | |-- similarity.txt
| | uttr_similarity
| | |-- simcse_100.txt
| multiwoz
| |-- all.csv
| | intent_info
| | |-- actions.json
| | |-- concepts.json
| | | descriptions
| | | |-- d1_pattern.json
| | | |-- names.json
| | | intent_similarity
| | | | simcse
| | | | |-- raws.json
| | | | |-- similarity.txt
| | uttr_similarity
| | |-- simcse_100.txt
python classification/train.py dataset={dataname} experiment.name=/path/to/experiment/dir
python classification/evaluate.py dataset={dataname} experiment.name=/path/to/experiment/dir
Specific setups
The default hyper-parameters settings to reproduce experiments for a specific dataset are detailed in the corresponding documentation:
./classfication/conf/dataset/{dataname}.yaml
You can replicate the experiment by running the bash command ./run_{dataname}.sh.
conf
|-- config.yaml
| dataset
| |-- atis.yaml
| |-- bank.yaml
| |-- clinc.yaml
| |-- multiwoz.yaml
| Parameter | Default | Description |
|---|---|---|
| dataset | ||
| dataset.name | Dataset and it's config name | |
| dataset.path | Relative path to split data or whole dataset | |
| dataset.intent_info_path | Relative path to intent information data | |
| dataset.description_type | Type of intent description to use. Ex: names, d1_pattern |
|
| dataset.uttr_len | Max length of utterance in tokens. Longer utterance would be truncated. | |
| dataset.desc_len | Max length of intent description in tokens. Longer utterance would be truncated. | |
| model | ||
| model.base_model | roberta-base | Contextualized encoder model name or path |
| model.dropout | 0.5 | Linear classifier head dropout |
| model.embedding_dim | 768 | Contextualized encoder embedding size |
| experiment | ||
| experiment.root_dir | ./ | Root path for experiments |
| experiment.name | ??? | Experiment name - needs to specify |
| experiment.seed | 0 | Random seed |
| experiment.epochs | Epochs to train | |
| experiment.batch_size | Batch size | |
| experiment.accum_steps | Number of gradient accumulation steps | |
| experiment.k_negative | 7 | Number of examples for negative sampling |
| experiment.train_only_seen | True | Train only with seen intent descriptions or not. |
| experiment.intent_desc_first | false | Is intent description above utterance in sentence pair encoding |
| experiment.test_epoch | None | Specify epoch for evaluation. Default: best loss epoch |
| experiment.temperature | 0.5 | Temperature for feature space contrastive learning |
| experiment.mlm_percent | 0.2 | The proportion of tokens masked in the sentence |
| experiment.embedding_param | 0.3 | the trade-off hyperparameter 𝜆 |
| experiment.mlm_param | 1 | the trade-off hyperparameter 𝜇 |
| scheduler | ||
| scheduler.lr | 2e-5 | Learning rate |
| scheduler.warmup_steps | 0.15 | Scheduler warmup iterations |
| checkpoint | ||
| checkpoint.save_from_epoch | None | Specified epoch to save checkpoint from. Default: save only best loss checkpoint |
| checkpoint.saved_model | None | Epoch to load model checkpoint from. Default: load from best loss checkpoint. |
| log.print_every | 1000 | Number of iterations to log loss. |