Better stacktraces in ioda

[rhoneyager-tomorrow]

Description

Here's a natural follow-on to my oops PR (JCSDA/oops#24). It inserts stack traces into the ioda-engines exceptions. These traces are immensely useful when debugging ObsSpace issues. Instead of receiving vague errors about ioda being unable to read a variable, if you're lucky (i.e. you have the right build flags and operating system / libstdc++ / libbacktrace / addr2line support) you now can see a list of stack frames with line numbers showing exactly where errors are occurring.

Here's what I mean:

Exception: level 0
	source_column:	0
	source_filename:	/backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/src/ioda/Engines/ObsStore/Variables.cpp
	source_function:	std::shared_ptr<ioda::ObsStore::Variable> ioda::ObsStore::Variable::read(gsl::span<char>, const ioda::ObsStore::Type &, ioda::ObsStore::Selection &, ioda::ObsStore::Selection &)
	source_line:	146
	stacktrace:	
 0# util::stacktrace_current[abi:cxx11]() at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/util/Stacktrace.cc:23
 1# ioda::Exception::add_call_stack() at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/src/ioda/Exception.cpp:46
 2# ioda::Exception::Exception(char const*, ioda::detail::compat::source_location::source_location const&, ioda::Options const&) at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/src/ioda/Exception.cpp:25
 3# ioda::ObsStore::Variable::read(gsl::span<char>, ioda::ObsStore::Type const&, ioda::ObsStore::Selection&, ioda::ObsStore::Selection&) at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/src/ioda/Engines/ObsStore/Variables.cpp:146
 4# ioda::Engines::ObsStore::ObsStore_Variable_Backend::read(gsl::span<char>, ioda::Type const&, ioda::Selection const&, ioda::Selection const&) const at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/src/ioda/Engines/ObsStore/ObsStore-variables.cpp:234
 5# ioda::detail::Variable_Base<ioda::Variable>::read(gsl::span<char>, ioda::Type const&, ioda::Selection const&, ioda::Selection const&) const at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/src/ioda/Variable.cpp:421
 6# ioda::Variable ioda::detail::Variable_Base<ioda::Variable>::read<int, ioda::detail::Object_Accessor_Regular<int, int>, ioda::Types::GetType_Wrapper<int, 0> >(gsl::span<int>, ioda::Selection const&, ioda::Selection const&) const at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/include/ioda/Variables/Variable.h:534
 7# ioda::Variable ioda::detail::Variable_Base<ioda::Variable>::read<int, ioda::detail::Object_Accessor_Regular<int, int>, ioda::Types::GetType_Wrapper<int, 0> >(std::vector<int, std::allocator<int> >&, ioda::Selection const&, ioda::Selection const&) const at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/engines/ioda/include/ioda/Variables/Variable.h:562
 8# void ioda::ObsSpace::loadVar<int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> >&, bool) const at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/ObsSpace.cc:821
 9# obsspace_get_int32_f at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/core/obsspace_f.cc:128
10# obsspace_mod_mp_obsspace_get_db_int32_ at /backup/ryan/src/jedi-testing/2024-02-backports/ioda/src/obsspace_mod.F90:327
11# ufo_crtm_utils_mod_mp_load_geom_data_ at /backup/ryan/src/jedi-testing/2024-02-backports/ufo/src/ufo/operators/crtm/ufo_crtm_utils_mod.F90:1023
12# ufo_radiancecrtm_mod_mp_ufo_radiancecrtm_simobs_ at /backup/ryan/src/jedi-testing/2024-02-backports/ufo/src/ufo/operators/crtm/ufo_radiancecrtm_mod.F90:385
13# ufo_radiancecrtm_simobs_f90 at /backup/ryan/src/jedi-testing/2024-02-backports/ufo/src/ufo/operators/crtm/ObsRadianceCRTM.interface.F90:106
14# ufo::ObsRadianceCRTM::simulateObs(ufo::GeoVaLs const&, ioda::ObsVector&, ufo::ObsDiagnostics&) const at /backup/ryan/src/jedi-testing/2024-02-backports/ufo/src/ufo/operators/crtm/ObsRadianceCRTM.cc:53
15# ufo::ObsOperator::simulateObs(ufo::GeoVaLs const&, ioda::ObsVector&, ufo::ObsBias const&, ioda::ObsVector&, ufo::ObsDiagnostics&) const at /backup/ryan/src/jedi-testing/2024-02-backports/ufo/src/ufo/ObsOperator.cc:53
16# oops::ObsOperator<ufo::ObsTraits>::simulateObs(oops::GeoVaLs<ufo::ObsTraits> const&, oops::ObsVector<ufo::ObsTraits>&, oops::ObsAuxControl<ufo::ObsTraits> const&, oops::ObsVector<ufo::ObsTraits>&, oops::ObsDiagnostics<ufo::ObsTraits>&) const at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/interface/ObsOperator.h:160
17# oops::Observer<fv3jedi::Traits, ufo::ObsTraits>::finalize(oops::ObsVector<ufo::ObsTraits>&) at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/base/Observer.h:226
18# oops::Observers<fv3jedi::Traits, ufo::ObsTraits>::finalize(oops::Observations<ufo::ObsTraits>&) at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/base/Observers.h:158
19# oops::CostJo<fv3jedi::Traits, ufo::ObsTraits>::computeCost() at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/assimilation/CostJo.h:187
20# oops::CostFunction<fv3jedi::Traits, ufo::ObsTraits>::evaluate(oops::ControlVariable<fv3jedi::Traits, ufo::ObsTraits>&, eckit::Configuration const&, oops::PostProcessor<oops::State<fv3jedi::Traits> >) at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/assimilation/CostFunction.h:250
21# int oops::IncrementalAssimilation<fv3jedi::Traits, ufo::ObsTraits>(oops::ControlVariable<fv3jedi::Traits, ufo::ObsTraits>&, oops::CostFunction<fv3jedi::Traits, ufo::ObsTraits>&, eckit::Configuration const&) at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/assimilation/IncrementalAssimilation.h:68
22# oops::Variational<fv3jedi::Traits, ufo::ObsTraits>::execute(eckit::Configuration const&, bool) const at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/runs/Variational.h:82
23# oops::Run::execute(oops::Application const&, eckit::mpi::Comm const&) at /backup/ryan/src/jedi-testing/2024-02-backports/oops/src/oops/runs/Run.cc:186
24# main at /backup/ryan/src/jedi-testing/2024-02-backports/fv3-jedi/src/mains/fv3jediVar.cc:25
25# __libc_start_call_main in /lib64/libc.so.6
26# __libc_start_main_alias_1 in /lib64/libc.so.6
27# _start in /backup/ryan/src/jedi-testing/2024-02-backports/build/bin/fv3jedi_var.x

I've also added in some binding functions to bring hdf5's exceptions into std::exception. See the hdf5_error_check() function. On any hdf5 failure, this call will read the hdf5 exception stack and re-throw it as a series of nested C++ exceptions. Implementing this throughout ioda would need many small one-line changes, and that would be a bit tedious to review, so it's best done in a separate PR.

Dependencies

List the other PRs that this PR is dependent on:

• Exception stack support and stacktrace improvements oops#24

Impact

Better error messages are their own reward. Downstream developers will

Checklist

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have run the unit tests before creating the PR

[srherbener]

Thanks for fixing this! I think the code looks good, just have the comments about the commented code.

[srherbener]

Thanks for removing the commented code!

[ctgh]

Thanks, this looks very useful!

[climbfuji]

Now that this PR has two approvals, I'll bring it over to JCSDA-internal for final review and merging.

[climbfuji]

PR is https://github.com/JCSDA-internal/ioda/pull/1224 - I will close this PR once https://github.com/JCSDA-internal/ioda/pull/1224 is merged.

[climbfuji]

Replaced by https://github.com/JCSDA-internal/ioda/pull/1224