Perhaps we could divide into multiple partitions, do indexing on its own in parallel, and combine the retrieval results based on the scoring? 👀 I guess the problem is more inherent to the particular type of indexing required on my end. Each document would take roughly 512 unique tokens, the vocabulary size is about 373K, and there are 8M documents.

8,841,823 isn't that msmarco? Given 340M tokens in total (using the bm25s.tokenize), it should be 34 tokens per doc on avg. So not sure if you are encountering a different issue in that case. However, I still recommend trying out your scenario with kaggle first to see if you are observing something different from my notebook; after all, parallelizing 4x 400it/s could at most get you 1600 it/s, which is far from the 18K/s processed on kaggle (very old cpu, run on single core).

Regarding the suggestion for parallelization, the problem is that the document frequency needs to be computed first before computing the rest, otherwise a certain level of parallelization could work, but are still reliant on the computing doc frequency first. Also, i have found that the native python multiprocessing seems to struggle with multithreading, achieving only ~2x improvement when going from 1 to 4 cores, so it might not be that useful.