After setting fp_task_min, e.g. =20, the fp data for one iter is less than 20 and training step is skiped. However, after several iters, these fp data are many and want to include them. Is there a setting to do that?

[roger13231]

As described in title. I have several iterations and each of them have around 10-15 fp data. The accurate is > 99% and fp data is few, so the training step is skipped which is expected. However, after several iterations, there are many of them and want to include them.

I found that in later training step (which is not skipped), these fp data is not included. I tried to maually change the input.json in one training step, but it goes back to normal (without these fp data) in the later iteration training step.

I am wondering whether there is a way to include them in later iterations.

[njzjz]

these fp data is not included

This is not expected. Are you sure iter.*/data.* exists?

In the code, all previous iterations should be included.

dpgen/dpgen/generator/run.py

Lines 394 to 398 in 47a23b3

	for ii in range(iter_index):
	if ii == iter_index - 1:
	old_range = len(init_data_sys)
	fp_path = os.path.join(make_iter_name(ii), fp_name)
	fp_data_sys = glob.glob(os.path.join(fp_path, "data.*"))

[roger13231]

Hi, Jinzhe,

Thanks for you rapid reply. The iter./data. exists.

I attached a few figures. fp_task_min is set to 20, and max is set to 80.

1. iter68 fp have system5 more than 20. The others have data less than 20;
2. Iter68 02.fp folder have data.003-007, 015, and 016. I checked these folder, all data folder have 'set.' folder with unempty data;
3. Iter62. All system fp data is less than 20..
4. Iter62. All 02.fp folder has data. folder having 'set.' folder wtih unempty data;
5. In the input.json file for training step in iter69, iter68 have only system5 data, and no iter62.

So, it seems like even data folder is generated, if the data is less than fp_task_min, the training does not include these data.

[njzjz]

Okay I found the related codes:

dpgen/dpgen/generator/run.py

Lines 418 to 422 in 47a23b3

	if nframes < fp_task_min:
	log_task(
	"nframes (%d) in data sys %s is too small, skip" % (nframes, jj)
	)
	continue

which was added in 37e59f8

@wanghan-iapcm could you clarify this behavior? It's confusing that these data was calculated but not used. By the way, this behavior is out of my expectations.