Use dqflags defined in roman_datamodels

[WilliamJamieson]

This PR updates romancal to take advantage of the changes in spacetelescope/roman_datamodels#293.

Checklist

• added entry in CHANGES.rst under the corresponding subsection
• updated relevant tests
• updated relevant documentation
• updated relevant milestone(s)
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below. How to run regression tests on a PR

[codecov]

Codecov Report

Attention: Patch coverage is 94.73684% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 76.78%. Comparing base (50663a2) to head (4233f35).
Report is 2 commits behind head on main.

❗ Current head 4233f35 differs from pull request most recent head d9a6459. Consider uploading reports for the commit d9a6459 to get more accurate results

Files	Patch %	Lines
romancal/lib/basic_utils.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1099      +/-   ##
==========================================
- Coverage   76.81%   76.78%   -0.04%     
==========================================
  Files         105      105              
  Lines        7048     7047       -1     
==========================================
- Hits         5414     5411       -3     
- Misses       1634     1636       +2     

Flag	Coverage Δ		*Carryforward flag
nightly	62.80% <ø> (-0.19%)	⬇️	Carriedforward from 50663a2

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[WilliamJamieson]

This does not effect the regression tests in any way, see: https://plwishmaster.stsci.edu:8081/job/RT/job/Roman-Developers-Pull-Requests/600/. All that has happened is the dqflag definitions have been moved to roman_datamodels.

[schlafly]

This looks good to me. It looks like 0.14.0 is in "draft" state, but we could merge after that actually comes out?

[ddavis-stsci]

I don't understand why we need this change. In general the dq flags are tightly coupled with the calibration process and it make sense to keep these together.
This also seems to be counter to most other calibration software that keeps the dq flags and calibration software together.
So what problem is this solving?

[schlafly]

Here are a couple of use cases:

• the simulator wants to populate DQ flags. It needs to know what the meanings of the bits are but doesn't need the rest of the pipeline. Presently we hack the few we need in, but getting these through the existing roman_datamodels dependency would be cleaner.
  https://github.com/spacetelescope/romanisim/blob/main/romanisim/parameters.py#L87-L88
• A user is processing our image. They already have roman_datamodels in order to open the files. They want to understand what the DQ flags mean, but don't want to bring in the full pipeline as a dependency.

[ddavis-stsci]

I'm not sure why you want to single out the dq values? You already put in add hoc values that are defined elsewhere, e.g. the read pattern is defined in the PRD. Are you also proposing that be added to the data models?

By moving the dq flags to roman_datamodels then you must have roman_datamodels for the cal code. In general this is not a problem but if someone (SSC?) wants to write a custom i/o package to read their data them you are still requiring them to install roman_datamodels to get the pipeline to work. Note: we do have a requirement to support custom data processing from the users (SOC-1061)

For a user to understand the dq flags we have a page specifically for to address this (https://roman-pipeline.readthedocs.io/en/latest/roman/dq_init/reference_files.html)

The cal pipeline is also the place where these values are "defined", The dqflags.py is really only a dictionary to allow you to use the names in place of a value. The real definition for these, what they do and how they are used, is encoded in the cal pipeline code. So I'm not sure that the dq flags should be separate from the cal code.

[schlafly]

I do not want to propose that we have read_pattern defined in roman_datamodels.

I see roman_datamodels as something that lets people read and interact with data we produce. It's natural for the DQ flags to live there since otherwise to be able to use names for the DQ bits you need to copy the definitions from the documentation. I agree that you can do that; the point of this is to allow people to avoid making those copies.

I also agree that there's another approach where rad, rdm, and romancal are one package. But insofar as we have decided to separate ~metadata and structures from algorithms, the names of the dq bits look to me to fit better with the metadata. Certainly the simulator would use these, and I anticipate that other code would as well.

I agree that other people will need to install some dependency in order to get to the flag definitions. The statement is just that roman_datamodels is a smaller dependency than the full pipeline, and that it's one that many users of our data products will want to install, since its whole purpose is to allow people to interact with the data files we produce.

[PaulHuwe]

This passes regtests: https://plwishmaster.stsci.edu:8081/job/RT/job/Roman-Developers-Pull-Requests/617/

[PaulHuwe]

LGTM

[schlafly]

LGTM, merging.