PEST-IES runtime issues (PLPROC, Factors-files, etc.)

[DStrom1987]

Hi,

I'm running a model with ~5k parameters. The model in itself takes about 30 seconds to run (since I have tuned down the convergence-criteria and made some other optimizations).

The model is in total 15 layers and has ~34 000 cells per layer. Total active cells is around 450 - 500 k. Since it seems like PEST makes a factors file for each parameter type and each layer, the export of factors-files takes quite some time. It takes significantly longer to export the factors-files for each worker run, around 2,5 - 3 minutes export vs 30 seconds of model run time.

When I'm running a large number of realizations this becomes a problem. Is there some better way of handling this? Are the factor-files updating each iterations to factor in the update to each parameter, making it necessary to export new ones every time the model is run, or could this step be cut out of the process?

The model doesn't have any prior data conflict and seems like it should be able to reach calibration targets quite easily. However it seems like i need a couple of thousand realizations to achieve an adequate fit. Should i then require a re-run for whatever reason, I'm looking at another couple of days to the process. Say, for example I'm running 10 000 reals * 3 minutes across 27 cores. It would take me around 18 hours to finish up one ensemble run, then close to the same time for upgrades (given that I don't drop any reals along the way). Three iterations would take up to 3 days with a possibility of having to rerun it afterwards.

Any advice on how to speed things up a bit? (I have run prior monte-carlo to see if the parameter ranges seem reasonable to get a desired level of fit).

Sincerely

[cnicol-gwlogic]

If you're talking about kriging factors, for example via plproc, then no you do not (typically#) need to update those per realization. You can make one set for your base model, and make sure it is in each worker directory on launch - but note that your plproc commands will need to change (remove / comment out the calc_kriging_factors parts, and just leave the actual krig commands operative. Make sure you remove any delete commands that may have been placed in your workflow batch.

'# the only time you may need to update those is for example if you were to adjust kriging parameters (angles, aniso, ranges etc) during model inversion. But that poses it's own issues with covariance etc, and would be a very rare case. As far as I know anyway.

[DStrom1987]

Thanks!
I tried to remove them from the batch file but it didnt really have an impact on the run time. Limited at best.

I dont really see the logic in exporting the files over and over for each pest run but I must be missing something. This is when running the forward run script generated in model muse and I suppose it isnt really optimized for running ensembles. Seems like I would have a quicker access to the underlying files without the plproc script with something like the forward run scripts in the gmdsi tutorials but since I have generated my tpl-files elsewhere I just want to pass them to pestpp-ies through the instructions in my .pst-file.

Any ideas on how to avoid the steps in the lower picture (taken from the run-model batch-file). Where I simply instruct pest to use whatever tpl-files, factors-files etc that are in the directory and are cited in the control file? The first picture gives some hints on whats going on with the PLPROC-script, which takes alot of time since it seems to re-generate the factors-files etc. every iteration.

[cnicol-gwlogic]

You can't remove those lines because they do the interpolation of params to mesh. Within those plproc.script files however, if the kriging factors commands are active, they will slow your run down unnecessarily. If you inactivated those in there (or they were not written there by modelmuse in the first place) and it is still too slow, then you are pretty much stuck and need more CPUs - your model sounds far quicker than most that I've seen.