Train GRU4Rec with large file

[anilkumargautam1]

HI

I have large atomic dile of interactions appox 10 GB, i am trying to train model with following code
queue = mp.get_context("spawn").SimpleQueue()
run_recbole(model='GRU4Rec', dataset='cd-data-2', config_file_list=['test_seq.yaml'],queue=queue)

but after some time its terminated with following message
Process finished with exit code 137 (interrupted by signal 9: SIGKILL)

Please suggest what i can do to train with large data set

Thanks
Anil

[arseny239]

Hi Anil
I am not a member of Recbole command, but did you check the memory consumption during the training? Maybe you are out of memory?
I also work with a large dataset (8Gb .item file, 800M .inter file). I did not use GRU4Rec, but I tried to use SASRec and SHAN.
Try to decrease the train_batch_size and eval_batch_size parameters in your .yaml file
Also try to consider the load_col parameter - maybe you don't need some features.

Sincerely yours,
Arseny