Minimalistic code for the training of DistilSPLADE_max models using the sentence-bert framework. We provide a file for training Splade_max(train_max.py) and one for training distilSplade(train_distill.py). Distillation is done using the latest training data released by Nils Reimer (https://twitter.com/Nils_Reimers/status/1435544757388857345/photo/1). Note that this is not exactly the code used in our papers, and that we are not yet able to provide indexing/retrieve code (but one could maybe use the beir-eval for that), however results are very competitive with state of the art and with what we observed with splade before.

MRR, R@1k and NDCG numbers are multiplied by 100 for simplicity of presentation. For the full result table please look at the end of this README

There are some differences in this training code to the one we used in the SpladeV2 paper namely:

• For SpladeMax:
  • Instead of BM25 negatives we use the ones from: https://twitter.com/Nils_Reimers/status/1435544757388857345/photo/1
  • We also use their strategy to remove hard negatives that maybe are not negatives at all (depending on the score of a cross attention model)
  • Training hyperparameters (epoch, lr, warmup...)
• For DistilSplade
  • Instead of DistilSplade negatives we use the ones from: https://twitter.com/Nils_Reimers/status/1435544757388857345/photo/1
  • Instead of using our cross attention scores, we use their negative scores
  • Training hyperparameters (epoch, lr, warmup...)
• Base networks
  • In the paper we always use distilbert-base-uncased
  • In this experiments the base network is explicitly added

For ensembles, scores are normalized following pyserini --normalization:

ensemble_score = sum_over_models ((score_model - (min_score_model max_score_model)/2) / (max_score_model-min_score_model))