RuntimeError: Job a6df06ad-b422-4aa4-9b19-9408f0a1a40c failed for more than 3 times

[huangwh23]

HELP！
Hi, everyone. When I run code dpgen run , it always gives this error. How do I fix it? PLEASE!!! The error message is as follows：

[AnguseZhang]

The error log tells that the job a6df06ad-xxxx has failed. At this time, you should enter the path where this task executes. It should be in one folder in work_path (you specify this in machine.json). Enter that folder and you will see a script file, such as xxxx.sub and some log files. You should see why this jobs fails from error log, and you can also try manually submit the file, and finally fix the problem.

[huangwh23]

Thank you!!!

[ChesterCs-thu]

I am facing the same problem. I checked the input.json and xxxx.sub files in work path and the output showed the error below:
/var/spool/torque/mom_priv/jobs/134135.admin.SC: line 16: /usr/bin/touch: Argument list too long
/var/spool/torque/mom_priv/jobs/134135.admin.SC: line 17: /usr/bin/touch: Argument list too long
/var/spool/torque/mom_priv/jobs/134135.admin.SC: line 30: /usr/bin/touch: Argument list too long
/var/spool/torque/mom_priv/jobs/134135.admin.SC: line 31: /usr/bin/touch: Argument list too long
/var/spool/torque/mom_priv/jobs/134135.admin.SC: line 40: /usr/bin/touch: Argument list too long

And when i manually submit the dp train input.json, it can work normally. But when i submit the xxxx.sub file, it shows the same errors. I want to know how to fix the problem.

[AnguseZhang]

Could you please post the full information of onexxx.sub ?

[SHANG-Ming-Hui]

I am currently experiencing the same problem. Can you diagnose the specific cause?
The problem occurs when the fp-tasks are submiting(task 07).
With in the fp.log, there is some information about the memory can not be allocated as
"[node81:28105:0] lb lface.c:665 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory
Abort(1615503) on node 37 (rank 37 in comm 0): Fatal error in PMPI Init: 0ther MPI error, error stack:
MPIR Init thread( 138)........:
MPID Init(1169)......................:
MPIDI OFI mpi init hook(1807):
create endpoint(2473)........:
0FI endpoint open failed (ofi init.c:2473:create endpoint:Input/output error)"

The xxx.sub file is posted as below.

---

---

#!/bin/bash
cd task.000.000000
test $? -ne 0 && exit 1
{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }
cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000001
test $? -ne 0 && exit 1
{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }
cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000002
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000003
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000004
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000005
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000006
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000007
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000008
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000009
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000010
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000011
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000012
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000013
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000014
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000015
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000016
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000017
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000018
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1
cd task.000.000019
test $? -ne 0 && exit 1

{ if [ ! -f tag_0_finished ] ;then
cp /public/home/smh/tasks/ordinary/DpMd/MLFF_class/DeePMD/dp-gen/KPOINTS ./ && source /public/software/profile.d/mpi_intelmpi-2021.3.0.sh && mpirun -np 40 /public/software/apps/vasp/intelmpi/6.3.0/bin/vasp_std && if [ $(grep NELMIN OUTCAR | wc -l) -eq 2 ];then sed -i '0,/NELMIN/s/NELMIN/TMP/' OUTCAR;fi 1>> fp.log 2>> fp.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }

cd /public/home/smh/tasks/ordinary/DpMd/MLFF_class/init_MD/dpgen/tmp/fp/83dd17ab-0b93-4326-b0b1-2ce06bd4d459
test $? -ne 0 && exit 1

wait

touch 83dd17ab-0b93-4326-b0b1-2ce06bd4d459_tag_finished

---

---

[zhanghuang103031]

I had the same problem, repeatedly submitting RuntimeError: Job: 9b65aa9c199cd023196be654948f0a7cee94a8ed 76904failed 3times. * * Error in `/home/Haichao/ANACONDA3/envs/deepmd/bin/Python' : Double Free O R corruption (out) : 0x00002b312c11b940 * * , look at the debug file and find RuntimeError: Failed to detect availbe GPUs due to: 2024-10-2906:27:28.129500: I tensorflow/core/platform/CPU. CC: 193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512 FMATO enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024.10-2906:27:28.616975: W TensorFlow/stream/Cuda/Cuda. CC: 374] a non-primary context 0x5ba7720 for device 0exists before initializing the streamexecutor. The primary context is now 0x2b792b8d0752. We Haven' t verified StreamExecutor works with that.2024-10-2906:27:28.617183: F tensorflowcore platform statusor.cc : 33] Attempting to fetch value instead of handling error INTERNAL: failed initializing StreamExecutor for CUDA device ordinal 0: INTERNAL: failed call to cuDevicePrimaryCtxRetain: Cuda: out of memory; total memory reported: 12627148800