This repository is the official implementation of TraceNet

To install requirements:

pip install -r requirements.txt

• SST-5 dataset , or from here to down sentence-level SST-5 dataset
• Download Yelp-5 dataset, and then sample data by sample_yelp_data.py file.
• glove.840B.300d is needed to generate sst5.hdf5 file and yelp5.hdf5 file. Since glove.840B.300d is about 5GB, we recommend you use SST5.glove.to.TraceNet under dataset/SST_5, which is extracted from the former according to SST-5 vocabulary.
• SentiWordNet_3.0.0.txt is need for evaluation on attacks.
• The source code of CNN, LSTM, BiLSTM we modified is from here
• The source code of Gumble Tree LSTM we modified is from here

To train the model(s) in the paper, run this command (SST-5 dataset for example). Note that, each command and coresponded results are included in log.sst5 file:

run CNN (rand, static, nonstatic, multichannel), LSTM, BiLSTM, under CNN_LSTM dir: first generate sst5.hdf5: python sst_process.py and then: python main.py --dataset ../dataset/SST_5/sst5.hdf5 --model_type KIMCNN --embedding_type rand --num_classes 5 --batch_size 50 --max_sent 49 --num_epochs 20 --n_hidden 300

run GT-LSTM rand : python train.py --task sst-5 --word-dim 300 --hidden-dim 300 --clf-hidden-dim 1024 --clf-num-layers 1 --dropout 0.5 --batch-size 64 --max_seq_length 49 --max-epoch 20 --pretrained False --save-dir save/ --device cuda

run Bert, XLNet, Roberta bert: python run_sentiment_classifier.py --do_train --do_eval --output_dir sst_bert_5307/ --model_type bert --model_name_or_path ../bert_base_en --per_gpu_eval_batch_size 1000 --task_name sst-5 --data_dir ../dataset/SST_5/ --num_train_epochs 10.0 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 32 --max_seq_length 128 --learning_rate 5e-5 --seed 1

xlnet: python run_sentiment_classifier.py --do_train --do_eval --output_dir sst_xlnet_5533/ --overwrite_output_dir --model_type xlnet --model_name_or_path ../xlnet_base_cased --per_gpu_eval_batch_size 1000 --task_name sst-5 --data_dir ../dataset/SST_5/ --num_train_epochs 10.0 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 16 --max_seq_length 64 --learning_rate 2e-5 --seed 1

roberta: python run_sentiment_classifier.py --do_train --do_eval --output_dir sst_roberta/ --model_type roberta --model_name_or_path roberta-base --do_lower_case --per_gpu_eval_batch_size 512 --task_name sst-5 --data_dir ../dataset/sst5/ --num_train_epochs 10.0 --logging_steps 300 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 16 --max_seq_length 128 --learning_rate 2e-5 --seed 42

roberta-MR: python run_sentiment_classifier.py --do_train --do_eval --output_dir mr_roberta/ --model_type roberta --model_name_or_path roberta-base --do_lower_case --per_gpu_eval_batch_size 512 --task_name mr-2 --data_dir ../dataset/mr/ --num_train_epochs 10.0 --logging_steps 300 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 16 --max_seq_length 128 --learning_rate 2e-5 --seed 42

roberta-sst2: python run_sentiment_classifier.py --do_train --do_eval --output_dir sst2_roberta/ --model_type roberta --model_name_or_path roberta-base --do_lower_case --per_gpu_eval_batch_size 512 --task_name sst-2 --data_dir ../dataset/sst2/ --num_train_epochs 10.0 --logging_steps 300 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 16 --max_seq_length 128 --learning_rate 2e-5 --seed 42

roberta-yelp: python run_sentiment_classifier.py --do_train --do_eval --output_dir yelp_roberta/ --model_type roberta --model_name_or_path roberta-base --do_lower_case --per_gpu_eval_batch_size 1024 --task_name yelp-5 --data_dir ../dataset/yelp5/ --num_train_epochs 10.0 --logging_steps 300 --dropout_prob 0.1 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 64 --max_seq_length 256 --learning_rate 2e-5 --seed 42

roberta-laptop: python run_sentiment_classifier.py --do_train --do_eval --output_dir laptop_roberta/ --model_type roberta --model_name_or_path roberta-base --do_lower_case --per_gpu_eval_batch_size 512 --task_name laptop --data_dir ../dataset/laptop/ --num_train_epochs 5.0 --logging_steps 50 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 16 --max_seq_length 128 --learning_rate 2e-5 --seed 42

roberta-restaurants: python run_sentiment_classifier.py --do_train --do_eval --output_dir restaurants_roberta/ --model_type roberta --model_name_or_path roberta-base --do_lower_case --per_gpu_eval_batch_size 512 --task_name restaurants --data_dir ../dataset/restaurants/ --num_train_epochs 5.0 --logging_steps 50 --dropout_prob 0.0 --output_feature 0 --num_hubo_layers 3 --method null --seq_select_prob 0.0 --per_gpu_train_batch_size 16 --max_seq_length 128 --learning_rate 2e-5 --seed 42

run TraceNet XLNet/roberta/glove TraceNet X: python run_sentiment_classifier.py --model_type xlnet_tracenet --model_name_or_path ../xlnet_base_cased --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 500 --overwrite_output_dir --output_dir sst_xlnetTraceNet_5642/ --task_name sst-5 --data_dir ../dataset/SST_5/ --num_hubo_layers 3 --method 'mean' --proactive_masking --seq_select_prob 0.2 --dropout_prob 0.3 --output_feature 128 --per_gpu_train_batch_size 16 --max_seq_length 64 --learning_rate 2e-5 --num_train_epochs 10.0 --seed 1

TraceNet R: python run_sentiment_classifier.py --model_type roberta_tracenet --model_name_or_path roberta-base --do_lower_case --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 512 --output_dir sst_robertaTraceNet/ --task_name sst-5 --data_dir ../dataset/sst5/ --num_hubo_layers 3 --method mean --proactive_masking --seq_select_prob 0.3 --dropout_prob 0.1 --output_feature 512 --per_gpu_train_batch_size 16 --max_seq_length 128 --weight_decay 0.0 --adam_epsilon 1e-6 --learning_rate 2e-5 --num_train_epochs 10.0 --logging_steps 300 --seed 42

TraceNet R mr: python run_sentiment_classifier.py --model_type roberta_tracenet --model_name_or_path roberta-base --do_lower_case --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 512 --output_dir mr_robertaTraceNet/ --task_name mr-2 --data_dir ../dataset/mr/ --num_hubo_layers 3 --method mean --proactive_masking --seq_select_prob 0.3 --dropout_prob 0.1 --output_feature 512 --per_gpu_train_batch_size 16 --max_seq_length 128 --weight_decay 0.0 --adam_epsilon 1e-6 --learning_rate 2e-5 --num_train_epochs 10.0 --logging_steps 300 --seed 42

TraceNet R sst2: python run_sentiment_classifier.py --model_type roberta_tracenet --model_name_or_path roberta-base --do_lower_case --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 512 --output_dir sst2_robertaTraceNet/ --task_name sst-2 --data_dir ../dataset/sst2/ --num_hubo_layers 3 --method mean --proactive_masking --seq_select_prob 0.3 --dropout_prob 0.1 --output_feature 512 --per_gpu_train_batch_size 16 --max_seq_length 128 --weight_decay 0.0 --adam_epsilon 1e-6 --learning_rate 2e-5 --num_train_epochs 10.0 --logging_steps 300 --seed 42

TraceNet R yelp: python run_sentiment_classifier.py --model_type roberta_tracenet --model_name_or_path roberta-base --do_lower_case --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 512 --output_dir yelp_robertaTraceNet/ --task_name yelp-5 --data_dir ../dataset/yelp5/ --num_hubo_layers 3 --method mean --proactive_masking --seq_select_prob 0.3 --dropout_prob 0.1 --output_feature 512 --per_gpu_train_batch_size 64 --max_seq_length 256 --weight_decay 0.0 --adam_epsilon 1e-6 --learning_rate 2e-5 --num_train_epochs 10.0 --logging_steps 300 --seed 42

TraceNet R laptop: python run_sentiment_classifier.py --model_type roberta_tracenet --model_name_or_path roberta-base --do_lower_case --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 256 --output_dir laptop_robertaTraceNet/ --task_name laptop --data_dir ../dataset/laptop/ --num_hubo_layers 3 --method mean --proactive_masking --seq_select_prob 0.3 --dropout_prob 0.1 --output_feature 512 --per_gpu_train_batch_size 16 --max_seq_length 128 --weight_decay 0.0 --adam_epsilon 1e-6 --learning_rate 2e-5 --num_train_epochs 5.0 --logging_steps 50 --seed 42

TraceNet R restaurants: python run_sentiment_classifier.py --model_type roberta_tracenet --model_name_or_path roberta-base --do_lower_case --do_train --do_eval --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 256 --output_dir restaurants_robertaTraceNet/ --task_name restaurants --data_dir ../dataset/restaurants/ --num_hubo_layers 3 --method mean --proactive_masking --seq_select_prob 0.3 --dropout_prob 0.1 --output_feature 768 --per_gpu_train_batch_size 16 --max_seq_length 128 --weight_decay 0.0 --adam_epsilon 1e-6 --learning_rate 2e-5 --num_train_epochs 5.0 --logging_steps 50 --seed 42

TraceNet G: python run_gpt_input.py --output_hidden_states --output_item_weights --per_gpu_eval_batch_size 16 --per_gpu_train_batch_size 16 --max_seq_length 49 --learning_rate 0.001 --output_feature 50 --dropout_prob 0.2 --num_train_epochs 10 --task sst-5 --weight_decay 0.2 --proactive_masking --seq_select_prob 0.05 --seed 42

To evaluate TraceNet-X (attacks) on SST-5, run:

generate attacks data from sst test set: python create_GA_test_sst.py and then python inference.py --output_dir sst_xlnet_5533/ --task_name sst-5 --data_dir ../dataset/SST_5/against/ --do_eval --per_gpu_eval_batch_size 1000 --model_type xlnet --model_name_or_path ../xlnet_base_cased --max_seq_length 64 --dropout_prob 0.0 --output_feature 0 --against --overwrite_cache or python inference.py --output_dir sst_xlnetTraceNet_5642/ --task_name sst-5 --data_dir ../dataset/SST_5/against/ --do_eval --per_gpu_eval_batch_size 1000 --model_type xlnet_tracenet --model_name_or_path ../xlnet_base_cased --max_seq_length 64 --dropout_prob 0.0 --output_feature 128 --against --overwrite_cache or python inference.py --output_dir sst_roberta_5669/ --task_name sst-5 --data_dir ../dataset/SST_5/against/ --do_eval --per_gpu_eval_batch_size 1000 --model_type roberta --model_name_or_path ../roberta_base_en --max_seq_length 128 --dropout_prob 0.0 --output_feature 0 --against --overwrite_cache or python inference.py --output_dir sst_robertaTraceNet_5787/ --task_name sst-5 --data_dir ../dataset/SST_5/against/ --do_eval --per_gpu_eval_batch_size 1000 --model_type roberta_tracenet --model_name_or_path ../roberta_base_en --max_seq_length 128 --dropout_prob 0.0 --output_feature 512 --against --overwrite_cache

• The transformer based pretrained language models we use are Bert (base, uncased), XLNet (base, cased) and RoBerta (base). With Transformers, you can download it by set model.from_pretrained('bert-base-uncased'), model.from_pretrained('xlnet-base-cased') and model.from_pretrained('roberta-base'), respectively.
• For models trained by our methods and finetuned on SST-5 dataset, you can download from here to inferece.

The following is the performance of baselines and TraceNet on SST-5 test set, for detailed parameters and results output, please refer to log.sst5 file: