2156 RIQ Definition of QC parameters

[hugobuddel]

Alain Smette writes in https://jira.eso.org/browse/MET-2156

This RIX is a follow-up on https://jira.eso.org/browse/MET-1472 (corresponding AI: https://jira.eso.org/browse/MET-1669), itself a follow-up on PDR ASM-28.

From the relevant top level requirements & specifications, one can derive (flow-down) the Quality Control Parameters that should be monitored in order to assess the instrument performance. An advanced draft document providing these parameters as well as sensitive definition of the corresponding thresholds and values should be done as part of the FDR; final version can of course only be completed by the end of the commissioning.

The library design document currently misses a number of QC parameters to monitor the instrument performance. The definition of a number of other parameters do not match the 'expected accuracy' in terms of quantity (the parameters provide 'absolute value' which cannot be directly compared with the 'expected accuracy' which are often defined as relative quantities).

SciOps and myself are available for support on this work. I provide in attachment a typical format that existing VLT instruments use.

[hugobuddel]

This RIX refers to an action item from the System FDR: https://jira.eso.org/browse/MET-1669

Based on a RIX of System FDR: https://jira.eso.org/browse/MET-1472

It also provides a template for QC parameters: TemplateForQCDocument.docx

I was unaware of this System FDR Action Item and also unaware of this template.

[hugobuddel]

Sent to @rvboekel and @Rumpelstil :

Hi Kieran, Roy, Wolfgang,

Could you help us reply to the RIQ we got about the "definition of the QC parameters"?

The RIX can be found on jira: https://jira.eso.org/browse/MET-2156

The text is:

...

The primary reason I'd like to involve you is that this is a follow-up on a System-FDR Action Item assigned to Roy. I was unaware of this Action Item, and have thus not taken it into account when writing the DRLD. (I now realize I should have been more proactive in checking the System-FDR AIs.)

The RIX does not directly asks us to do anything, but the following items can be distilled:

• There should be a flow-down from the top-level requirements to QC parameters to assess the instrument performance. Does this flow-down exist, and if not, who should make it?
• There should apparently be a document describing the thresholds and similar properties of these QC parameters. While that makes sense, we are not aware that such a document is to be delivered at FDR; it is not listed in ESO-037611. There is a TemplateForQCDocument.docx attached to the RIX, which I have not seen before. Who should write this instrument-monitoring document?
• Apparently the "library design document currently misses a number of QC parameters to monitor the instrument performance". That could very well be, our focus was on QC parameters to assess the data quality (and did a mediocre job at that, see RIX MET-2147 to MET-2155). Could you help us define more QC parameters to monitor the instrument?
• Furthermore, the "definition of a number of other parameters do not match the 'expected accuracy' in terms of quantity". That would indeed be our mistake. We claim that we use QC parameters to assess whether the data meets expected accuracies, but apparently did not define them in a way that makes this possible; we can take the lead on that.

Summarizing the above: the Data Reduction Library should produce the QC parameters relevant for instrument monitoring, and describing these parameters in the DRLD is therefore the responsibility of the Pipeline team (so the RIX is correct in identifying that these parameters are missing). However, instrument monitoring itself, and thus deciding which QC parameters are required for monitoring, is (I think) someone else's responsibility, so your assistance in defining the parameters would be appreciated.

I will reassign MET-2156 to Kieran for now.

Thanks for your help,

Hugo

[Rumpelstil]

My main FDR AI related to QCs was to associate specific QC parameters with the individual calibration data products (update 4.1 to the calibration plan, which is a METIS-internal release. Version 5 will be the PAE version).

We also discussed QCs with ESO in the METIS Science Operations Telecon on June 6, 2023 to address MET-1669. ESO stated that there is no official list of required QCs. We then discussed the reasonably required QCs for METIS (also inspired by CRIRES, VISIR, ERIS, etc.).

The TemplateForQCDocument.docx, which I'm be made aware of for the 1st time today, looks useful. We should agree on a schedule for filling such a "technical report"(?).

[hugobuddel]

Answered with

The QC parameters in the DRLD are currently mostly focused on data quality, and indeed less so on instrument health monitoring (although many can be used for both). We will update the document to include more instrument monitoring parameters (and more QC parameters in general).

We are already in the process of collaborating with ESO to identify and define these parameters. We appreciate sharing TemplateForQCDocument.docx; the structure given in the template is useful to capture the necessary information.

Note that the listed 'expected accuracies' in the DRLD refer refer to the recipes, and are used to develop/test/validate the recipe (and to communicate to users what accuracy to expect from the recipes). These accuracies 'precede' performance/instrument monitoring; e.g. a QC parameter can be measured accurately, and still have a value outside acceptable limits.

It seems that we were expected to include the requested information in the Calibration Plan. ESO-037611 v4 states (line 1208):

1208 The document includes the two main sections:

1. Scientific data calibrations: [...]
2. Instrument monitoring: [...]

Most of the CP is dedicated to the former, and the latter seems missing. It would be my preference to include part of the information (also) in the DRLD, in particular the acceptable limits on the QC parameters, as this would make the DRLD more self-contained.

I do think the (current) PIP team can define sensible QC parameters for instrument health monitoring. But I'm not sure we are qualified to define sensible limits on these parameters.

We also just didn't have enough time to define enough QC parameters, we got several RIXes on that. When allocating our time, I focused on dependencies between the teams within the consortium. We only took QC parameters for data validation into account, and this is mostly within the boundaries of PIP. I would have given the QC parameters more attention had I realized that instrument health/perfomance monitoring is perhaps more important, because that involves multiple teams.

[hugobuddel]

The QC parameters were briefly discussed at the system team meetings. I sent the following email to several people (first to Olivier, Wim, Roy, Gilles, Kieran, Felix, Adrian, then also to Wolfgang Brandner, Chad, Kora, Thomas Bertram, and Yannis)

Attached is a template document from ESO that we can use to describe
how we want to use quality control (QC) parameters for instrument
health monitoring. It contains a table template that we can use to
describe for each relevant FITS keyword, how it is measured, its
nominal/minimum/maximum values, etc.

The document was shared with us in the context of an FDR RIX on the
Data Reduction Library Design (DRLD, E-REP-AST-MET-1006) assigned to
me, see MET-2156 [1]. However, the DRLD should only contain details on
how parameters are derived from the data, or how parameters already in
the raw data are propagated. That technical information is indeed
missing from the DRLD so we will add that to close the RIX.

Information about which monitoring parameters should exist, and how
they should be used to assess the health of the instrument, is beyond
the scope of software design, and thus should be described in a
separate document. The attached template seems suitable, and describes
what information ESO ultimately expects from us. I therefore propose
that we, as a consortium, will use that template (or similar) to
collect all the required health monitoring QC parameters.

If you agree on creating this new health monitoring document, then the
question would be who takes the lead on creating and maintaining it. I
believe this effort can best be led by someone who will either do
(part of) the monitoring themselves, or by someone who has the
decisive power on what range of instrument conditions is acceptable.

A bit of history is worth noting. The "Dataflow for ESO Observatories
Deliverables Standard" (ESO-037611 v4, or the "611 / 1618 document")
states that the Calibration Plan (CP, E-PLA-NOVA-MET-1066) should have
two main sections, one on "scientific data calibrations", and one on
"instrument monitoring":

2. Instrument monitoring: This section describes which instrument calibration data has to be
   collected with which frequency to allow trend analysis of the instrument health and to initiate
   preventive maintenance.

However, this second section was not included in the Calibration Plan.
In fact, the CP does not include any QC parameters at all, and I
(later) understood that the idea was to define the QC parameters in
the DRLD instead of in the CP, see also RIX MET-898 [2].

The PIP team has taken the responsibility for defining QC parameters
for scientific data quality assessment, because we are close to the
data. However, in my opinion, the PIP team should not be responsible
for defining the instrument health monitoring QC parameters, because
we are distant from the instrument.

Cheers,

Hugo

[1] RIX on DRLD with attached document: https://jira.eso.org/browse/MET-2156
[2] RIX on CP about QC parameters: https://jira.eso.org/browse/MET-898

[hugobuddel]

We discussed the QC params for monitoring at the Ops meeting. Sent this message afterwards:

@ Wolfgang, could you perhaps share the CRIRES calibration plan for reference?

My takeaway from the meeting is that there are many different types of
quality control / health monitoring / safety monitoring / performance
monitoring parameters.

From PIP we are mostly interested in the parameters that need to be
derived by processing them, because we'd need to write the code to do
that. And we got RIXed because we did not define any of those
parameters explicitly. The chopper offset we discussed might be such
an example. We expect it to be stable, but need to verify it indeed
is, and potentially take action if it drifts too much. Please share
more of those parameters with us if you have them.

The main other category seems to be parameters measured directly by
one of the subsystems and that don't really have a need to go into the
FITS headers. These parameters do not really involve the pipeline
team. Perhaps we could still use a document like the template provided
by ESO to also describe those parameters.

Edit: sorry Wolfgang, in the email I was referring to Wolfgang Brandner but now I accidentally tagged you

[hugobuddel]

Perhaps we should just add a few of these parameters to the document following the scheme sent to us by Alain Smette and take it from there.

[hugobuddel]

Attached is the ESO-281271_5 CRIRES Instrument Calibration Plan.pdf. This does list some QC parameters for monitoring, but not in the template format from ESO.

Example:

However, it was not directly evident from the code of cr2res_cal_wave what QC parameter is actually used to monitor the lamp intensities.

[hugobuddel]

As sent to Olivier, Rofy, Wim, Felix, Adrian, Wolfgang B.,

After some discussion within our team and also with ESO, we decided
that it is probably the easiest to include all QC parameters for
instrument health monitoring in the Data Reduction Library Design, and
also in the FITS headers. Even those parameters that are derived by (a
subsystem of) the instrument, because it is nice to have everything in
the DRLD, and having information in the FITS headers is useful for
data mining.

ESO asked us to provide the information about
health-monitoring-parameters in a different form (a table) than for
data-assessment-parameters. So we decided to add a new section to the
QC parameters section dedicated to instrument health monitoring. At
least for now. We took a possible QC parameter that Olivier proposed
in another thread, and ran with it. Attached you can see how it looks.
The table layout and the legend is taken from the
TemplateForQCDocument. On overleaf [1] or github [2] you can see the
whole document.

However, and this is where you come in, it should not be the PIP team
that defines these parameters and their limits. So if you want to use
any parameter for instrument health monitoring, please send them in!

You can provide the parameters in any form that is convenient to you.
But beware: we might need to make up information if it is missing. The
most convenient form for us would be in the form at the bottom of the
attached TeX file. Or you can also add the parameters directly to the
document on overleaf or github. We will probably add some more
parameters ourselves, with limits "TBD at AIV or commissioning", but
those will probably be boring ones like "median dark current value"
and such. We need you for the interesting ones.

Attached was page 270-272 from #288 (now merged)

[hugobuddel]

We should split the discussion of the health monitoring QC parameters into two parts:

1. What is required to close the AI and pass FDR.
2. What is best for METIS and the consortium.

We should focus on passing FDR, so I propose

• Define some more parameters in the right template, e.g. for dark value, maximum number of bad pixels, etc, enough to close the AI. Just best guessing those limits.
• Close RIX
• Continue the discussion on how to define the parameters and there limits within the consortium.

[hugobuddel]

Slides by Mario van den Ancker on 2024/04/18
Slides_METIS_SciOps_18.04.2024.pdf