Calculate the BM25 ranking matrix, MTM/preprocess/BM25/data/bm25_scores_(#Claims, #Articles).npy
, by MTM/preprocess/BM25/Weibo.ipynb
.
MTM/dataset/Weibo/splits/data_splits.ipynb
- Split by event
- Split by claim
MTM/preprocess/tokenize/run.sh
MTM/preprocess/ROT/run.sh
- Prepare RougeBert's Training data
- Training
- Vectorize the claims and articles (get embeddings)
- Prepare the clustering data,
MTM/preprocess/PMB/data/Weibo/clustering_training_data_[TS_SMALL]<dist<[TS_LARGE].pkl
, byMTM/preprocess/PMB/calculate_init_thresholds.ipynb
. - Kmeans clustering:
MTM/preprocess/PMB/run.sh
- See some cases of key sentences selection:
MTM/preprocess/PMB/key_sentences_selection_cases.ipynb
MTM/model/run.sh