some errors in call "dpgen run pa... ma..."

[hjh19970102]

Dear developers,
When I call "dpgen run pa... ma...",it have something wrong about call"lmp_mpi",the error is as following:(it concludes 10 initial configurations).
*** Error in /public3/home/sc31148/a/lammps/src/lmp_mpi': double free or corruption (!prev): 0x00000000031eb0e0 *** INFO:dpgen:job 6ec96f6e-e2f0-4832-8e5e-2d6db31ba8a0 terminated, submit again INFO:dpgen:job e6f52053-7806-4467-808d-dd28b357af50 terminated, submit again INFO:dpgen:job 5253a482-fc5a-434f-a0ab-132921c4f5c3 terminated, submit again INFO:dpgen:job 8c3a9990-fb1e-473e-9851-7a07dc6e50e0 terminated, submit again INFO:dpgen:job f18139e0-f8ca-4edd-8ce4-30b2b4b1c920 terminated, submit again *** Error in /public3/home/sc31148/a/lammps/src/lmp_mpi': double free or corruption (fasttop): 0x00000000028f9870 ***
INFO:dpgen:job 2c277044-bc55-4ad5-abf3-e1a0475d15ea terminated, submit again
INFO:dpgen:job d55941a2-8ca3-46fd-b043-e53c6d4aafee terminated, submit again
INFO:dpgen:job e0bc2f24-0589-4350-b96e-c057e14161cb terminated, submit again
INFO:dpgen:job de00f61b-2311-4606-b5fa-cff57cd1e53d terminated, submit again
INFO:dpgen:job 6ec96f6e-e2f0-4832-8e5e-2d6db31ba8a0 terminated, submit again
INFO:dpgen:job e6f52053-7806-4467-808d-dd28b357af50 terminated, submit again
INFO:dpgen:job d4a20368-ffda-490c-ae14-d75122d50bcb terminated, submit again
*** Error in /public3/home/sc31148/a/lammps/src/lmp_mpi': double free or corruption (fasttop): 0x00000000027ae850 *** INFO:dpgen:job f18139e0-f8ca-4edd-8ce4-30b2b4b1c920 terminated, submit again INFO:dpgen:job 5253a482-fc5a-434f-a0ab-132921c4f5c3 terminated, submit again INFO:dpgen:job 2c277044-bc55-4ad5-abf3-e1a0475d15ea terminated, submit again INFO:dpgen:job 8c3a9990-fb1e-473e-9851-7a07dc6e50e0 terminated, submit again INFO:dpgen:job de00f61b-2311-4606-b5fa-cff57cd1e53d terminated, submit again INFO:dpgen:job 6ec96f6e-e2f0-4832-8e5e-2d6db31ba8a0 terminated, submit again INFO:dpgen:job d55941a2-8ca3-46fd-b043-e53c6d4aafee terminated, submit again INFO:dpgen:job e0bc2f24-0589-4350-b96e-c057e14161cb terminated, submit again *** Error in /public3/home/sc31148/a/lammps/src/lmp_mpi': double free or corruption (!prev): 0x00000000021507f0 ***
*** Error in /public3/home/sc31148/a/lammps/src/lmp_mpi': double free or corruption (!prev): 0x00000000017fc7f0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x81299)[0x2b21ffbf1299] /lib64/libc.so.6(+0x39d10)[0x2b21ffba9d10] /lib64/libc.so.6(+0x39d37)[0x2b21ffba9d37] /public1/soft/gcc/4.9.2//lib64/libgomp.so.1(+0x6dc6)[0x2b21fff44dc6] /public1/soft/gcc/4.9.2//lib64/libgomp.so.1(+0xeaff)[0x2b21fff4caff] /public1/soft/gcc/4.9.2//lib64/libgomp.so.1(GOMP_parallel+0x3a)[0x2b21fff478ba] /public1/home/sc31148/a/deepmd_root/lib/libdeepmd_op.so(_ZN15ProdVirialSeAOp7ComputeEPN10tensorflow15OpKernelContextE+0x8ab)[0x2b21f145fcfb] INFO:dpgen:job e6f52053-7806-4467-808d-dd28b357af50 terminated, submit again INFO:dpgen:job f18139e0-f8ca-4edd-8ce4-30b2b4b1c920 terminated, submit again INFO:dpgen:job 5253a482-fc5a-434f-a0ab-132921c4f5c3 terminated, submit again INFO:dpgen:job d4a20368-ffda-490c-ae14-d75122d50bcb terminated, submit again *** Error in /public3/home/sc31148/a/lammps/src/lmp_mpi': double free or corruption (!prev): 0x00000000020fd4c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81299)[0x2ac6adac4299]
/lib64/libc.so.6(+0x39ce9)[0x2ac6ada7cce9]
/lib64/libc.so.6(+0x39d37)[0x2ac6ada7cd37]
/public1/soft/gcc/4.9.2//lib64/libgomp.so.1(+0x6dc6)[0x2ac6ade17dc6]
/public1/soft/gcc/4.9.2//lib64/libgomp.so.1(+0xeaff)[0x2ac6ade1faff]
/public1/soft/gcc/4.9.2//lib64/libgomp.so.1(GOMP_parallel+0x3a)[0x2ac6ade1a8ba]
/public1/home/sc31148/a/deepmd_root/lib/libdeepmd_op.so(_ZN15ProdVirialSeAOp7ComputeEPN10tensorflow15OpKernelContextE+0x8ab)[0x2ac69f332cfb]
INFO:dpgen:job de00f61b-2311-4606-b5fa-cff57cd1e53d terminated, submit again
Traceback (most recent call last):
File "/public3/home/sc31148/.local/bin/dpgen", line 11, in
load_entry_point('dpgen==0.8.0', 'console_scripts', 'dpgen')()
File "/public3/home/sc31148/.local/lib/python3.7/site-packages/dpgen/main.py", line 182, in main
args.func(args)
File "/public3/home/sc31148/.local/lib/python3.7/site-packages/dpgen/generator/run.py", line 2309, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/public3/home/sc31148/.local/lib/python3.7/site-packages/dpgen/generator/run.py", line 2284, in run_iter
run_model_devi (ii, jdata, mdata)
File "/public3/home/sc31148/.local/lib/python3.7/site-packages/dpgen/generator/run.py", line 1010, in run_model_devi
errlog = 'model_devi.log')
File "/public3/home/sc31148/.local/lib/python3.7/site-packages/dpgen/dispatcher/Dispatcher.py", line 91, in run_jobs
while not self.all_finished(job_handler, mark_failure) :
File "/public3/home/sc31148/.local/lib/python3.7/site-packages/dpgen/dispatcher/Dispatcher.py", line 215, in all_finished
raise RuntimeError('Job %s failed for more than 3 times' % job_uuid)
RuntimeError: Job 6ec96f6e-e2f0-4832-8e5e-2d6db31ba8a0 failed for more than 3 times
So.Could you tell me what should I do next?

[apachehigh5]

I am also having the same issue:

Description
------------
Traceback (most recent call last):
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/bin/dpgen", line 10, in
sys.exit(main())
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/dpgen/generator/run.py", line 2410, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/dpgen/generator/run.py", line 2385, in run_iter
run_model_devi (ii, jdata, mdata)
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/dpgen/generator/run.py", line 1059, in run_model_devi
dispatcher.run_jobs(mdata['model_devi_resources'],
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/dpgen/dispatcher/Dispatcher.py", line 91, in run_jobs
while not self.all_finished(job_handler, mark_failure) :
File "/home/jeet/Installs/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/dpgen/dispatcher/Dispatcher.py", line 216, in all_finished
raise RuntimeError('Job %s failed for more than 3 times' % job_uuid)
RuntimeError: Job 256806bb-4317-4034-b803-7bef8e364fa0 failed for more than 3 times

Would be extremely grateful if the developers can provide some help on this.

[njzjz]

You need to go to the working directory, i.e. 6ec96f6e-e2f0-4832-8e5e-2d6db31ba8a0, to see what happened.

@AnguseZhang DP-GEN should give a better message for this error.

[AnguseZhang]

OK

[huangwh23]

I am also having the same error. Is the problem solved, and how is it solved?

[AnguseZhang]

Hi @Kyrie-666 Please post your error log. I have answered your another problem in #544, and wonder if that solves all your problems.

[huangwh23]

Yes, thank you!

[Ketcd]

I encountered a similar issue when call "dpgen run param.json machine.json". During the model_devi tasks in LAMMPS, I received the error "double free or corruption (!prev) ". Do you have any suggestions on how I can continue the calculation for exploration?
handle_unexpected_job_state
raise RuntimeError(err_msg)
RuntimeError: job:d646becfdf5e3040b853fccde2c71122896d988f 31342 failed 3 times.
Possible remote error message: ==> /home/ecust/lws/7f0a5bf39d7e078c0e1d94027c48dbeaa5dd170c/task.000.000001/model_devi.log <==
5 min
Histogram: 1 0 0 0 0 2 0 0 0 1
Neighs: 0 ave 0 max 0 min
Histogram: 4 0 0 0 0 0 0 0 0 0
FullNghs: 5 ave 12 max 0 min
Histogram: 1 0 0 2 0 0 0 0 0 1

Total # of neighbors = 20
Ave neighs/atom = 4
Neighbor list builds = 11
Dangerous builds = 0
Total wall time: 0:00:08
double free or corruption (!prev)
double free or corruption (!prev)
double free or corruption (!prev)
double free or corruption (!prev)
/root/anaconda3/envs/deepmd/bin/lmp: line 11: 34812 Aborted (core dumped) /root/anaconda3/envs/deepmd/bin/_lmp "$@"
/root/anaconda3/envs/deepmd/bin/lmp: line 11: 34810 Aborted (core dumped) /root/anaconda3/envs/deepmd/bin/_lmp "$@"
/root/anaconda3/envs/deepmd/bin/lmp: line 11: 34811 Aborted (core dumped) /root/anaconda3/envs/deepmd/bin/_lmp "$@"
/root/anaconda3/envs/deepmd/bin/lmp: line 11: 34813 Aborted (core dumped) /root/anaconda3/envs/deepmd/bin/_lmp "$@"