How should we approach UQ for our projects? Standardization, Benchmarks, Software

[bnord]

@beckynevin @AleksCipri @voetberg @AeRabelais @onionesquereality @zlin7 @Pooja-G04

Major/Overarching questions for UQ in DeepSkies projects and infrastructure

1. How can we add UQ to our workflows that train, test, and diagnose our models?
   2. How much can this be automated?
   3. Should we incorporate anything into DeepUtils?
2. One aspect of our vision for DeepSkies is that we could have fully automated UQ applied for each project.
3. What are the benchmark methods for different tasks -- i.e., classification and regression?
4. What are the software out there that we might build on.
5. Should we consider domain adaptation in this conversation? It relates to systematic errors.

[bnord]

Some UQ codes

1. https://github.com/pypest/pyemu
2. https://github.com/zlin7/LVD
3. https://easyvvuq.readthedocs.io/en/dev/
4. https://www.sciencedirect.com/science/article/pii/S0010465597001070

[voetberg]

For me, the ideal is to have a two tier approach - something simply to look at immediately for an outsider (similar to a test coverage badge), and a more detailed automatic report that shows exactly where the model is strongest and weakest over various metrics.

The feasibility of something totally automated seems low, but semi-automated practices seem within reach and could easily be included in some sort of toolkit. Probably similar to the way utils has the auto-EDA module and auto-quality checks, a UQ-focused training loop or post training UQ report would have a place there.

I'm reading through these linked projects now to see if there's something good to poach.

[bnord]

Do you know where you would draw the line between semi-automated and fully-automated?

[voetberg]

I'd personally define fully as a single button press, generally post-hoc. You install the package and run a single command line thing that runs UQ, or have it included in a check/unit test. Semi automated, by my definition, requires some level of setup that needs you to know your project. It requires data formatting, adding things to workflows, and pipeline setup, or intervention before training to include it in your training loops. It's executed when the training loop is.

[bnord]

I imagine some post-hoc stuff being permitted to run fully with the warning that the user needs to customize it. Also, I think some methods are sometimes fully post-hoc -- e.g., deep ensembles is almost fully post-hoc.

I think I mostly agree about what you said for semi-automatic.