Wikikgv2 Model Training is Hanging

[HarryShomer]

Hi,

I'm having an issue where the model gets stuck while training. It typically happens early in the first epoch. Below is an example when running the smore/training/vec_scripts/train_shallow_wikikgv2.sh (unmodified sans the GPUs) on 4 NVIDIA RTX A6000 50GB GPUs.

It hangs forever unless I stop it with a keyboard interrupt. Doing so yields the following traceback (I only post a portion because it's very long and repetitive).

It seems like something is happening in the multiprocessing as it's hanging when sharing messages between processes.

Any help would be appreciated! @hyren

Thanks,
Harry

[hyren]

Hi, this does not seem to be related to the smore repo. Here is an instruction I found, can you please check?
https://www.rdmamojo.com/2012/05/18/libibverbs/

[HarryShomer]

That doesn't seem to be the issue.

I tried seting up libibverbs, and while the warnings went away, the code is still hanging.

[hyren]

Hi, can you say more about the details of the environment?

[HarryShomer]

Sure. Here's some basic info. Let me know if you would like anything else.

OS: Ubuntu 20.04.5 LTS
CUDA: 11.6.124
GPU(s): NVIDIA RTX A6000
Python: 3.9.12
PyTorch: 1.12.1

[Hanjun-Dai]

Hi there, sorry for the inconvenience, could you please quickly try with only 1 gpu, and see if it still hangs? We are trying to figure out whether it is due to gpu-gpu communication, or the c++ sampler we used.

[HarryShomer]

Sorry, I should have been clearer in my initial comment. The code doesn't hang when running with just one GPU. This only occurs when using multiple GPUs.

[Juanhui28]

We tried one gpu again, actually it still hangs. Sorry for the inconvenience.

[Hanjun-Dai]

Hi there,

Could you please kindly pull the latest code in the wikikgv2 branch, and add --train_async_rw=False to your script and try again? Since I'm unable to reproduce your issue, I'd like to see whether this would temporarily resolve the issue.

[Juanhui28]

Hi,

We tried your suggestion and now the training doesn't hang! But we got the similar issue in the evaluation. And we noticed your suggestion in another issue and follow the suggestion, but it still hangs.

I really appreciate your help!

[hyren]

hi, what's the script you used?

[Juanhui28]

Hi, train_shallow_wikikgv2.sh in the training/vec_scripts folder.

[hyren]

Just to make sure, you used the scripts in train_shallow_wikikgv2.sh and add the --train_async_rw=False flag?

[Juanhui28]

Actually no, since if we add it we got the unrecognized arguments error.

[hyren]

have you pulled the most recent commits?

[Juanhui28]

Sorry we made some mistakes when we merged the code. Now we pulled the the latest code and it works for both training and evaluation!

Thank you so much!