Hi I'm trying to train model on my custom data which is about 4 GB. Unfortunately we lack enough memory and preprocess phase kills. Is there any way that we can feed data in multiple batches to preprocess phase to generate vocab.txt and interaction.csv and then sum up them for training phase? (Because it seems that whole data should be generate once).
Thanks for your support.