Suspicious QP short test failures with anomalously large errors

[prckent]

Describe the bug

For the excited state case, the C4 multideterminant tests seem to be wrong in mixed precision. The ground state calculation looks more reliable, but there are still some large deviations. Possibly there is a bug here, possibly the test is not well enough designed and doesn't get reliable error bars. It looks like outlier energies can be obtained, indicating a possible bug.

I have been going through the short tests to "tame" them. Most failures would need only modest increase in error bar to pass, e.g. are failing at 3.1 sigma. These QP ones have noticeably much larger deviations.

@kgasperich , @anbenali ? Any insights/guesses?

short-C4_md-ae-excited-qp-vmc-1-16-totenergy
https://cdash.qmcpack.org/testSummary.php?project=1&name=short-C4_md-ae-excited-qp-vmc-1-16-totenergy&date=2024-09-19 (52 sigma failure for one of the gcc14-mixed builds!)

short-C4_md-ae-excited-qp-vmc-1-16-variance
https://cdash.qmcpack.org/testSummary.php?project=1&name=short-C4_md-ae-excited-qp-vmc-1-16-variance&date=2024-09-19

short-C4_md-ae-excited-qp-vmc-1-16-kinetic
https://cdash.qmcpack.org/testSummary.php?project=1&name=short-C4_md-ae-excited-qp-vmc-1-16-kinetic&date=2024-09-19

Ground state seems more reliable, but still some large deviations:

short-C4_md-ae-ground-qp-vmc-1-16-variance
https://cdash.qmcpack.org/testSummary.php?project=1&name=short-C4_md-ae-ground-qp-vmc-1-16-variance&date=2024-09-19

To Reproduce

Standard ctest runs with develop

Expected behavior

Test passes reliably.

System:
CPU systems

[kgasperich]

My initial suspicion is that this might just be some strange pathological case. Does anyone know how this came to be a test system and how the inputs were generated? (did they converge nicely, were there other nearby excited states, etc.)
If the wavefunctions are particularly bad, could this cause issues with ergodicity that create some strong dependence on the starting walker configurations?

[prckent]

We could look at the scalar files that are produced to see if there are wild outliers or a warmup problem. Maybe the initial guessed electron positions are poor, particularly for the excited state.

[prckent]

Increasing the warmup from 100 steps to 5000 was not successful, e.g.

3/14 Test #396: short-C4_md-ae-ground-qp-vmc-1-16-totenergy ....***Failed    0.07 sec
Tests for series 0
  Testing quantity: LocalEnergy
    reference mean value     : -151.05936896
    reference error bar      :   0.06753150
    computed  mean value     : -154.73220467
    computed  error bar      :   1.18603575
    pass tolerance           :   0.20259451  (  3.00000000 sigma)
    deviation from reference :  -3.67283571  (-54.38699880 sigma)
    error bar of deviation   :   1.18795677
    significance probability :   1.00000000  (gaussian statistics)
    status of this test      :   fail

[prckent]