-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve flagging and noise estimation #695
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to work on this PR a little more. There needs to be a distinction between samples that should not be used in template fitting and filtering and samples that are excluded from all processing. Point source flags are a good example of usable but excluded samples. We still want the point sources in the final map. I have flagged completely excluded samples such as turnarounds as "invalid" but now they seem to drift into the processing mask.
Summarizing a discussion from other channels: Principles
Implementation Plan
|
1184648
to
81254cb
Compare
… for disabled templates in template matrix.
…problems due to assumptions about looping over local detectors. Add extra debug plot support to offset template.
Ok @keskitalo , this PR is ready for re-review. There is a small issue when running on GPU with hybrid pipeline data movement, but that will likely just be a typo fix somewhere. Will not merge until those tests pass (which are not tested by CI anyway). |
…nits, even if not processing detectors from some observations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly small observations.
The one big question is about det_flag_mask
being repurposed and whether that is general enough.
…-sample flag mask
…reconditioner. Fix timing imports to point to timing rather than utils module.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!
We have two primary flag bits for shared and detector data:
"invalid" and "processing". The invalid bit means that the
data does not exist or is wrong / corrupted in some way that
makes it useless for any application. The processing bit
means that the data should not "typically" be used for
science results.
This work attempts to change all built-in simulation and
reduction operators to follow these basic guidelines:
solution exists
dropped packets, etc)
routinely be used for science, and these samples should
be flagged with the processing bit.
solutions or readout problems, no samples should be flagged
as invalid.
those features are typically known by low-level data
acquisition systems. Examples might be cooler cycles or
scan turnarounds, although those samples might also be
flagged by downstream operators.
passed through and remain invalid
this bit passed through.
that data is flagged as invalid, then the output samples of
the simulation operator should be flagged as invalid.
bit, it is up to the operator whether to use those samples.
those samples should be marked as invalid since they do not
contain the full expected detector response
those samples might be marked with the processing flag
passed through and remain invalid
this bit passed through as well.
the processing bit.
some samples with garbage, zeros, etc, then those samples
should be marked as invalid.
to use / process some data, then those samples should have
the processing bit set.
This PR also include improvements to flagging in the Offset template and a typo fix in fitting noise models.