AC-REFAC sensitivity and calibration

[Benjamin-Maglio]

Starting a discussion for new users of the AC-REFAC branch to assist with tutorials, debugging, and workflows.

[Benjamin-Maglio]

Some initial directions on post-processing of sensitivity analysis results in Jupyter Notebooks from a conversation with @jsclein-uaf.

For loading necessary packages etc:

import pandas as pd
import os
import sys
sys.path.insert(0, "/work/mads_calibration")
import SA_post_hoc_analysis 

I then load the data from SA like this:

os.chdir('/data/workflows/ac-cal-test/CMT31/CMAX_GPPAllIgnoringNitrogen/tobey_config')
param_props = pd.read_csv('param_props.csv')
sample_matrix = pd.read_csv("sample_matrix.csv")
targets = pd.read_csv('targets.csv', skiprows=1)
results = pd.read_csv('results.csv')

Where you'll need to update the path to your output directory in os.chdir()

Then you can call the analysis scripts like this:

SA_post_hoc_analysis.plot_spaghetti(results, targets)

USEFUL TIPS:

If you type:
SA_post_hoc_analysis.
and press the tab button on your keyboard it will show a list of functions you can use:

Then if you select one and type this in an empty cell:

SA_post_hoc_analysis.plot_spaghetti?

and run it, it will print out what you need to enter to get the function to work:

NOTE: there is currently a bug where the first time the analysis script is run it will expect a plots/ directory in your output directory. This will be resolved soon, but for now, you can get round this by running mkdir plots after you have used os.chdir() to move into your output directory.

To output ranges and initial guesses for MADS, we have been using these commands:

best_params, best_model = SA_post_hoc_analysis.n_top_runs(results, targets, sample_matrix, r2lim=0.9)
best_params

and then:

print(SA_post_hoc_analysis.prep_mads_distributions(best_params))
print(SA_post_hoc_analysis.prep_mads_initial_guess(best_params))

You may need to change the r2lim as before and check how many best_params you are getting, that's why I've been running these in two cells

[tobeycarman]

A few additional notes:

• If you set your Python path variable in your containing shell, you don't need the import sys; sys.path.insert(0, "/work/mads_calibration") lines at the top of all your notebooks. We should probably set the PYTHON_PATH in the docker containers, but haven't gotten to it yet. In the meantime, you can run the following when you sign into your container:

  export PYTHON_PATH="/work/mads_calibration:$PYTHON_PATH"

• We recently added a function that will load all 4 files at once. You can

  import SA_post_hoc_analysis as sap
  param_props, sample_matrix, targets, results = sap.load('/data/workflows/calibration/CMT06-IMNAVIAT/output/cmax_gppallignoringnitrogen/sa-N300/')

[valeriabriones]

Has anyone has nearly completed a calibration run only for the run to fail because of an error with the plot_optimization_fit? Specifically a trip up with the axes. Is there a way to call the plotting function post run in case this could happen? I don't think it was possible with the previous version of calibration and is unfortunate that results don't save if this occurs

[tobeycarman]

@valeriabriones there is not really an explicit way to do this yet.

It would be really nice, especially for testing so that you could work on pieces of the process without waiting for the time consumptive parts (the optimization loop, any other "forward" run of the model).

One thing you can do is comment out a few lines in AC-MADS-TEM.jl so that it doesn't run the optimization (the call to Mads.calibrate(..) - that is the function that starts the optimization. You might have to take an additional step to load the calibrated parameters from your .iterationresults file rather than using the data that are returned from the Mads.calibrate(...) call. There is a helper function in SA_post_hoc_analysis (yeah, not the best place to keep that function?) that will do this, there might be another function floating around that will parse those Mads files...

Another solution is to simply run the util.metrics.plot_optimizaiton_fit(...) function directly from Python (skipping all the Julia stuff). You would would have to load all the data into your Python session, but this might not be difficult...