Ability to load and add to PstFrom?

[wkitlasten]

The real selling point for PstFrom is the ability to easily parameterize and pestify models with heaps of pars and obs. I find my script crashing more than I like to admit and usually well into the script after a substantial amount of time/comp. When it crashes it requires starting over entirely due to "I/O error on closed file." It would be handy to be able to do it in steps, for starters the ability to do pars separate from obs... that is, save the pst file with pars only (maybe dummy obs for a test run), then load the same pst file into a PstFrom object to add observations. I imagine it would also be handy to load an existing PstFrom instance and add additional pars/obs using the bells and whistles in PstFrom.

Does this functionality exist and I am just missing it?

[briochh]

I guess you could try pickling the PstFrom object at interim points. I have no idea if it would load safely and efficiently but might be an easy fist thing to try.
Interested to hear if that works.

[jtwhite79]

You could also get closer to rolling your own with Pst.add_pars and Pst.add_obs after you get a basic interface done with PstFrom (not that PstFrom makes basic interfaces!)

[mnfienen]

Pickling should work - generally any python object can be pickled. The only bummer is there is a lack of portability like form OS to OS and Py version to version (so some degree). But...for the purposes of not losing progress in a single script to be picked up again by the same env, that should work fine I think

[briochh]

@wkitlasten, did you explore pickling option? I wonder if the logger calls might get in the way of this approach. Presumably, the logging file would likely be closed and the loaded pickle would not have knowledge to reopen the log file. Perhaps we could have a little method to relaunch a logfile (if existing) and attach to the reloaded PstFrom object to support further add_parameters() or add_observations() call etc.

Thoughts @jtwhite79?

[wkitlasten]

I tried pickling. I couldn't get it to work... but you know me! Has anyone else got it working?

I basically broke all my pf calls in to subroutines with a "use_pickle" kwarg. After setting up some preliminary bits I call: pf = build_pst_from(model, og_sim_ws=ogdir, template_ws=pdir, use_pickle=True):

def build_pst_from(model_name, gwf=False, og_sim_ws=False, template_ws=False,
                  use_pickle=False):
   if not og_sim_ws:
       og_sim_ws = os.path.join('d:\\', 'modelling', model_name)
   if not template_ws:
       template_ws = os.path.join('d:\\', 'modelling', model_name + '_temp')
   # initiate PstFrom
   pf = pyemu.utils.PstFrom(original_d=og_sim_ws, new_d=template_ws,
                            remove_existing=True,
                            longnames=True, spatial_reference=sr,
                            zero_based=False, start_datetime="1-1-2011")
   pf.extra_py_imports.append('flopy')
   if use_pickle:
       pickfile = open(os.path.join(template_ws, f"{model_name}.pkl"), 'wb')
       pickle.dump(pf, pickfile)
       pickfile.close()
       return
   else:
       return pf

2023-02-21 10:51:30.479894 starting: opening PstFrom.log for logging
2023-02-21 10:51:30.479894 starting PstFrom process
2023-02-21 10:51:30.479894 starting: setting up dirs
2023-02-21 10:51:30.488027 starting: copying original_d 'D:\modelling\wairau\20230111_wairau\wairau_from_glm' to new_d 'C:\wairau\wairau\wairau_14_s1_500_ies_pickle'
2023-02-21 10:51:30.834380 finished: copying original_d 'D:\modelling\wairau\20230111_wairau\wairau_from_glm' to new_d 'C:\wairau\wairau\wairau_14_s1_500_ies_pickle' took: 0:00:00.346353
2023-02-21 10:51:30.836418 finished: setting up dirs took: 0:00:00.356524
Traceback (most recent call last):
File "D:\modelling\nzmf6\nzmf6\build_pst_from.py", line 1026, in
pf = build_pst_from(model, og_sim_ws=ogdir, template_ws=pdir, use_pickle=True)
File "D:\modelling\nzmf6\nzmf6\build_pst_from.py", line 28, in build_pst_from
pickle.dump(pf, pickfile)
TypeError: cannot pickle '_io.TextIOWrapper' object

At which point I gave up and just suffer through multitasking while it re-builds every time I make a mistake (...you know me).

Anything obviously wrong there?

[briochh]

My suspicion would be that it might be something related to the open logger instance. We may need to handle that before and after pickling.