Spin asymmetric results from a spin symmetric wavefunction

[jtkrogel]

Describe the bug

QMCPACK produces spin-asymmetric results (as seen in the self-healing estimator) from a spin-symmetric wavefunction.

The system is AFM CoO (4 atom cell). A multideterminant wavefunction file with singles excitations in both up and down spins is being used. The coefficients are set with 1.0 on the ground state determinant and 1e-18 on all others to make the wavefunction equivalent to a single determinant run. In this case B-spline orbitals are used.

In other runs on graphene with gaussian orbitals, no issues have been observed.

Details of the estimator implementation
For a multideterminant wavefunction with orthogonal deteriminants ($\Phi_i$),
$\Psi =\sum_ic_i\Psi_i = e^J\sum_ic_i\Phi_i$

The self-healing estimator of coefficient $i$ is:
$c_i\propto\frac{\langle\Phi_i\vert e^{-J}\rvert \Psi\rangle}{\langle\Psi\lvert\Psi\rangle}$
(When $J=0$, proportionality is replaced by equality)

For a VMC run this can be rewritten as
$c_i\propto\frac{\langle\Psi\lvert e^{-2J}\frac{\Psi_i}{\Psi}\rvert\Psi\rangle}{\langle\Psi\lvert\Psi\rangle}$

The ratio $\Psi_i/\Psi$ can be found using a parameter derivative of the wavefunction as $\frac{\partial}{\partial c_i}\log\Psi$, so in general the estimator for VMC or DMC is
$c_i\propto\langle e^{-2J}\frac{\partial}{\partial c_i}\log\Psi\rangle$

This estimator is very simple, and for $J=0$ it only requires the parameter derivative to be correct.

Observed correctness of the estimator

For $J=0$ and $J\ne0$ this estimator has been tested in this case (CoO) and others, and the expectation value of the estimator in VMC always returns results statisically consistent with the inputted coefficients in the multideterimant.

Observed anomalous behavior
The measured coefficients from a DMC run break spin symmetry (note the large coefficients for up excitations at low index):

Only a subset of the coefficients with (4 sigma significance) are shown.

In addition, half of the estimated coefficients have miniscule stastitical fluctuations (5 orders of magnitude smaller than the other half) and have values near zero. This aspect (analyzed in more detail below) is the same in DMC or VMC, with or without a Jastrow.

The plot below shows the estimated coefficients from a VMC run, sorted in order of errorbar magnitude. The coefficients on the right have normal fluctuations, while those on the left are 5 orders of magnitude smaller. The vertical line marks 50%.

In an AFM state, like the one we use for CoO, each singles excitation in the up channel is equivalent to another in the down channel (also explicitly verified by exact matching of up and down Kohn-Sham eigenvalues in DFT).

Each such pair should produce identical sampling for the coefficient. However, many such pairs exist (roughly half of them) where the estimated coefficient for the up excitation has a finite fluctuation (errorbar) while for the down one a minute fluctuation is observed.

The plot below shows the statistical errorbar of up excitations (x-axis) with their corresponding down spin pairs (y-axis):

The normal and minute fluctuation coefficients show a highly structured pattern when organized by particle and hole indices (23 occupied orbitals in each spin channel, 207 virtuals in each):

The pattern is checkerboard-like in particle and hole indices. Here dark rectangles mark particle/hole (i,j) pairs having normal fluctuation in the estimated coefficient with those with miniscule fluctuations shown in yellow.

When similarly organized for the down excitations, a different checkerboard-like pattern emerges, again showing the spin asymmetry:

The source(s) of the anomalous spin asymmetry and the patterned minute fluctuations is unknown.

To Reproduce
File location: OLCF Frontier at /lustre/orion/mat151/proj-shared/qmcpack_bug_issue_5229

Files:

Co.opt.xml                                            qmc.s000.config.h5
einspline.tile_100010001.spin_0.tw_0.g0.bandinfo.dat  qmc.s000.cont.xml
einspline.tile_100010001.spin_1.tw_0.g0.bandinfo.dat  qmc.s000.random.h5
multidet_co2.h5                                       qmc.s000.scalar.dat
O.opt.xml                                             qmc.s000.stat.h5
pwscf.pwscf.h5                                        qmc.sbatch.in
qmc.info.xml                                          qmc.sbatch.in.e17654
qmc.in.xml                                            qmc.sbatch.in.o17654

The estimated coefficients are in the stat.h5 file:

>h5ls qmc.s000.stat.h5/sh_coeff
value                    Dataset {1000/Inf, 9523}

QMCPACK version:

  Git branch: develop
  Last git commit: 851f1bede1679efbcca3cfc810598f0eef100295
  Last git commit date: Fri Oct 18 10:18:32 2024 -0700
  Last git commit subject: Merge pull request #5204 from ye-luo/adjust-test

QMCPACK build: CPU complex (SplineC2C for orbitals)

 CPU only build
 Complex build. QMC_COMPLEX=ON

QMCPACK drivers: batched (legacy shows same issue)

Expected behavior
Spin symmetric results.

System:
Inti. AMD CPU machine at ORNL, 128 core nodes.

[prckent]

Checking in on the status of this since there were quite a few conversations and there are at least 4 areas that need investigating to narrow this down. While the patterns in the output strongly suggest some plumbing issues, there could also be numerical issues or API issues at play.

1. Is the VMC run correct? i.e. is the multi-determinant wave function being evaluated correctly?
2. Are splines different from periodic Gaussians/LCAO, and if so, how?
3. Are the results affected by precision issues?
4. Is self-healing operator implemented correctly/are the APIs it is using correct, if different from those used by VMC and DMC? That operator is relatively new code.

It is worth noting that a lot of changes were made to the multi-determinant code in the last year. Multideterminant molecular runs are supposedly OK per conversations with @anbenali , as well as usage and testing. These code paths are well tested and well used even for PBC LCAO ground state runs, e.g. the recent multideterminant lithium cobaltate runs would presumably show issues. However, there are some issues #5172 that could indicate bugs or numerical stability issues for excited states.

Are there any other possibilities to consider?

To address (2), if the tooling makes it easy enough, it would be good to run one of both of (1) Guassian AFM CoO (2) graphene with splines.

If any of these runs were mixed precision it would be good to verify that double precision does not change anything.

[jtkrogel]

I think I now know what is causing this behavior. I will have full confirmation after the Thanksgiving holidays.

Please no one put any time into this until I get back with the results.