Train.sh data loading stops at 70%

[Ariyaris]

Dear all,

After installing DiffDock-PP as per installation guide I attempted to run train.sh script.
Unfortunately loading of data stops somewhere at 70%. To be more precise, it becomes very slow and then dies.

Does anybody have advice or a workaround to this challenge?

Thank you!

Luka

[TernencezzZ]

Hi, @Ariyaris

I guess that there might be a memory overflow issue during ESM feature extraction. DIPSLoader computes embeds for all receptors and ligands from DIPS dataset in function compute_embeddings.

you can imagine how big it is:

Perhaps you can consider using some chunking methods to alleviate the memory pressure.

[onlyonewater]

hi, @TernencezzZ , could you reproduce the results with this code? if yes, can you tell me what parameters you used? like num_gpus, lr, ns, nv parameters.

[TernencezzZ]

hi, @onlyonewater

Actually no, due to the limit of my computer memory, I only used the first 8,000 data from the DIPS Dataset for training. I took the configuration in dips_esm.yaml and have only trained for around ten epochs. The loss is still decreasing.

[onlyonewater]

so do you use only one GPU? or use multi-GPU to train?

[Ariyaris]

Perhaps you can consider using some chunking methods to alleviate the memory pressure.

Hi @TernencezzZ

Do you have advice on how I could use chunking methods?
I only have 1 GPU and I am quite the novice in all this so any advice is helpfull.

Yours sincerely,
Ariyaris