Improve Albany debug output?

[ikalash]

This was an email chain b/w Albany developers a few weeks ago. Opening an issue so we don't forget about it. Here is the gist of the discussion, from @bartgol .

. They are indeed "debug" features. But I think we can do a bit better, to make them usable also by "not-super-expert-users", so to speak. I agree we can discuss them (I might have to miss the discussion since I go on vacation for 3 weeks starting monday).

Some thoughts (since I'll miss the discussion):
- The output should be written from our ModelEvaluator, since I believe it has more control and more knowledge of where we're at in the iteration. E.g., as I wrote before, computeGlobalResidual might never be called, if the residual is evaluated together with the Jacobian earlier.

- We should have a better way to express the will to print the solution. Currently, we specify an int in the yaml file, and -1 means "print all", 0 means "print none", and N>0 means "print only the N-th". This is not very clear, and btw, what if I only want to print the iteration 0? Perhaps we can have a more structured sublist like

Debug Output:
  Write Residual To MatrixMarket File:
     When: Iter # Valid options: All (every res), None (Default), Iter, BeginNoxStep, BeginNoxSolve, ...
     Iter: 3
     Prefix: my_res # Defaults to the usual 'rhs'
  Write Jacobian To MatrixMarket File: 
     When: None # None, All, BeginNoxSolve,...
     Prefix: my_jac
  Write Solution To MatrixMarket File:
     When: None # None, All, Last, Initial, BeginNoxSolve, ...
      Prefix: my_sol

"Iter: 3" should save only the 3rd residual, though I'm not sure how to interpret this: 3rd time that f was not a null ptr, regardless of whether it's a backtracking iter, or 3rd nox iter?

- Maybe some of these options (like BeginNoxSolve vs BeginNoxStep) are not possible to detect, so this might be too optimistic. However, I know for ROL we found a way to inject the optimization status into the parameter list, so that Albany could detect if we were at a new optimization iteration, or just a new NOX solver iteration within the same optimization iteration. Maybe we can hack Piro NOX solver to do the same (i.e., inject NOX iteration into the PL shared with the ME).

I am not sure how easy/quick all of this is. Perhaps super easy, perhaps hard. I think we could quickly implement a "verbose" format for the output PL (at least for the "All", "None", "Iter" options). Adding the logic to, say, skip backtracking iterations might be harder, or even not possible without modifying Piro...

[ikalash]

Lets talk about this at one of the Albany meetings after @bartgol returns from vacation.