QuasarNET afterburner only uses HIZ template results in overlap region

[dylanagreen]

When rerunning redrock the qos-qn afterburner runs the HIZ and LOZ templates separately due to the way RR accepts priors. The intention in the code is to then compare the two results in the overlap region, and keep the one with the lower chi^2 (this is normally done internally in redrock).

When running the two templates separately the afterburner initializes global variables:

desispec/py/desispec/scripts/qsoqn.py

Line 198 in e9ac854

redshift, err_redshift, chi2, coeffs = np.zeros(targetid.size), np.zeros(targetid.size), np.inf * np.ones(targetid.size), np.zeros((targetid.size, 10))

When aggregating the output the results are compared to the global variables, and if the fit has a better chi^2 than the globally stored chi^2 variable the fit is accepted and updated in the global variables:

desispec/py/desispec/scripts/qsoqn.py

Lines 251 to 256 in e9ac854

	# aggregate the result:
	best_chi2 = np.zeros(targetid.size, dtype='bool')
	best_chi2_tmp = chi2[sel] > chi2_tmp
	best_chi2[sel] = best_chi2_tmp
	redshift[best_chi2], err_redshift[best_chi2], coeffs[best_chi2] = redshift_tmp[best_chi2_tmp], err_redshift_tmp[best_chi2_tmp], coeffs_tmp[best_chi2_tmp]

However, the loop neglects to update the global chi2 variable, so line 253 above is always comparing against np.inf * np.ones(targetid.size) as initialized in line 198. The effect of this is that since the HIZ templates are run after the LOZ templates the HIZ results will always overwrite the LOZ results in the overlap region, regardless of whether the fit is better or at a different redshift than the LOZ templates.

This is probably a one line fix, adding

chi2[best_chi2] = chi2_tmp[best_chi2_tmp]

to update the global chi2 variable. after L256 should solve this issue, although validating this might take a little bit of effort. Reminder that the overlap region is 1.4 < z < 1.6.

[sbailey]

Good catch. An arguably better long term solution is to update rrdesi --templates to accept more than one template. Then the afterburner can call Redrock once, let Redrock sort out which HIZ/LOZ is best by its standard algorithm, and then the after burner just uses that. It feels like a lot of the complication / fragility here is that rrdesi currently does not allow you to specify more than one template unless you give it a full directory worth of templates (which then brings in templates that the afterburner doesn't want to use).

[abrodze]

Stephen's suggestion should already be possible since Redrock can take a text file for the --templates argument.

https://github.com/desihub/redrock/blob/main/py/redrock/templates.py#L298-307

I'll play around with the qnqso.py script to get it up to speed with this "new" redrock feature.

[sbailey]

Fixed by PR #2402; closing