Be/create ifu wkf

[eiseleb47]

I'd like to merge the IFU workflow skeleton into main as it is working as of now up till the calibrate_ifu_task.

I believe the issue stems from somewhere in in the metis_ifu_telluric recipe as the output of it creates an IFU_TELLURIC.fits with a FLUXCAL_TAB tag. Thus the calibrate task can't find a IFU_TELLURIC frame and fails.

However I've seen that there were mulitple changes done to the recipes on main, including the switch to tags. So I wanted to leave them alone on my branch for now and see if it works with the changes.

[sesquideus]

Nicely done! I can rebase main onto this branch and fix all (trivial) conflicts, then it passes all 248 and xfails all 10 tests.

The question is, should I?

[sesquideus]

When I tried to run edps -w metis.metis_lm_ifu_wkf -i $SOF_DATA and got

FileNotFoundError: could not resolve targets [] meta-targets ['science'] for workflow 'metis.metis_lm_ifu_wkf'

[hugobuddel]

Nice! Thanks @eiseleb47 .

I'm not sure I'm still qualified to review this properly :-). But looks fine too me.

I'm not sure whether these 'pro' classes like sci_reduce_ifu_class now allow us to use an existing file with PRO.CATG of IFU_SCI_REDUCED in the pipeline. At least I'm assuming that is the intent. But we can iterate on this later.

You can resolve @sesquideus remark about could not resolve targets by adding a 'science' meta target. See #88 on how to do so.

Perhaps you could import everything in metis_wkf.py?

[eiseleb47]

Thank you for the quick feedback from both of you!

The missing SCIENCE metatarget was indeed the reason. I have added it to the postprocess task and the cascade now executes as expected.

I have also added an import statement to the metis_wkf.py file to import the IFU workflow.

[hugobuddel]

In #90, I've branched from this branch and let github action run the full EDPS workflow, so also the IFU one, see this commit: ea6ac56

That failed because this branch is too far behind main, so I merged main into #90 , and now it does run all the IFU recipes in the github action, but most of them fail.

That would mean that the EDPS IFU workflow probably only works on your machine, and not on those of the rest of us.

Could you please either merge #90 into this branch, or otherwise merge main into this branch and manually make the change of ea6ac56 , and then make sure the workflow actually runs on github?

FWIW, here is the output of the test of #90 : https://github.com/AstarVienna/METIS_Pipeline/actions/runs/13102135125/job/36551697376

{'metis_det_dark': ['http://localhost:5000/jobs/d35aff7f-ccf7-45a3-92f7-66431f6599a6'],
 'metis_det_detlin': ['http://localhost:5000/jobs/7eaf3c10-12de-453f-8a56-1719dc6513d7'],
 'metis_ifu_calibrate': ['http://localhost:5000/jobs/455e7d37-5a6d-47e4-9233-fbcb03fad3fb'],
 'metis_ifu_dark': ['http://localhost:5000/jobs/da3ea8ff-8626-4567-9fd3-574db103f804'],
 'metis_ifu_distortion': ['http://localhost:5000/jobs/8028663e-c51c-4c00-92a8-ee6f5fa90414'],
 'metis_ifu_lingain': ['http://localhost:5000/jobs/859d99db-e37a-4945-9586-419738ff2110'],
 'metis_ifu_postprocess': ['http://localhost:5000/jobs/342881db-42ed-471f-a09b-442e61ab1a09'],
 'metis_ifu_rsrf': ['http://localhost:5000/jobs/8b7a9c8c-4c97-4283-b0e1-22e2f8cc95cf'],
 'metis_ifu_sci_reduce': ['http://localhost:5000/jobs/8257b540-e05e-47ae-a853-e370001e12ba'],
 'metis_ifu_sci_telluric': ['http://localhost:5000/jobs/034cb462-1f59-4c91-9edb-30617bf4cf82'],
 'metis_ifu_std_telluric': ['http://localhost:5000/jobs/ffe1edbf-2675-42c5-9dac-415c2a49dc3e'],
 'metis_ifu_wavecal': ['http://localhost:5000/jobs/0904d80a-6b85-45f0-93d4-79e709d66eb0'],
 'metis_lm_img_basic_reduce': ['http://localhost:5000/jobs/d3d9642e-8149-41ce-8846-7be7edff3af9'],
 'metis_lm_img_flat': ['http://localhost:5000/jobs/d251b8a3-9601-4972-9cb7-1457a8692aeb'],
 'metis_std_reduce': ['http://localhost:5000/jobs/cf54d062-21dd-4392-b26c-ba03972b6efa']}
00h00m00s {'PENDING': 14, 'RUNNING': 1} 
00h00m05s {'FAILED': 7, 'COMPLETED': 8} Failed jobs: 8b7a9c8c-4c97-4283-b0e1-22e2f8cc95cf, 034cb462-1f59-4c91-9edb-30617bf4cf82, 8257b540-e05e-47ae-a853-e370001e12ba, cf54d062-21dd-4392-b26c-ba03972b6efa, 342881db-42ed-471f-a09b-442e61ab1a09, 455e7d37-5a6d-47e4-9233-fbcb03fad3fb, ffe1edbf-2675-42c5-9dac-415c2a49dc3e
00h00m05s {'FAILED': 7, 'COMPLETED': 8} Failed jobs: 8b7a9c8c-4c97-4283-b0e1-22e2f8cc95cf, 034cb462-1f59-4c91-9edb-30617bf4cf82, 8257b540-e05e-47ae-a853-e370001e12ba, cf54d062-21dd-4392-b26c-ba03972b6efa, 342881db-42ed-471f-a09b-442e61ab1a09, 455e7d37-5a6d-47e4-9233-fbcb03fad3fb, ffe1edbf-2675-42c5-9dac-415c2a49dc3e
15 jobs processed in 00h00m05s
You can find reduced data, logs and quality control plots in the following directories:
=======================================================================================
/tmp/EDPS_data/METIS/metis_det_dark/d35aff7f-ccf7-45a3-92f7-66431f6599a6
/tmp/EDPS_data/METIS/metis_lm_img_flat/d251b8a3-9601-4972-9cb7-1457a8692aeb
/tmp/EDPS_data/METIS/metis_det_detlin/7eaf3c10-12de-453f-8a56-1719dc6513d7
/tmp/EDPS_data/METIS/metis_lm_img_basic_reduce/d3d9642e-8149-41ce-8846-7be7edff3af9
/tmp/EDPS_data/METIS/metis_ifu_lingain/859d99db-e37a-4945-9586-419738ff2110
/tmp/EDPS_data/METIS/metis_ifu_dark/da3ea8ff-8626-4567-9fd3-574db103f804
/tmp/EDPS_data/METIS/metis_ifu_distortion/8028663e-c51c-4c00-92a8-ee6f5fa90414
/tmp/EDPS_data/METIS/metis_ifu_wavecal/0904d80a-6b85-45f0-93d4-79e709d66eb0

So it tries to run all 11 IFU recipes, but only 4 succeed.

[sesquideus]

I managed to rebase main onto this and wrote a bunch of tests (mb/edps-tests). However, I got the same results, 8/15 working.

[eiseleb47]

I have merged #90 into the branch. The issue was in the rsrf task. It wasn't including a WCU_OFF_RAW file since it isn't mentioned in the DRLD, at least not the version I have from the DRLD repo.

I've added the class, data source and associated input. Additionally I forgot adding the sky raws to the sci and std tasks. Added those as well.

Lastly it still fails on two tasks, namely the calibrate and post process task. This is since the telluric recipe right now only produces a FLUXCAL_TAB.fits, but IFU_TELLURIC.fits is needed. The rest should now execute.

[hugobuddel]

I have merged #90 into the branch. The issue was in the rsrf task. It wasn't including a WCU_OFF_RAW file since it isn't mentioned in the DRLD, at least not the version I have from the DRLD repo.

It could very well be that either the recipe and/or the DRLD is wrong. Comparing with similar recipes and with the templates ( https://metis.strw.leidenuniv.nl/wiki/doku.php?id=operations:metis_templates ), I'm not sure what is right.

The most general thing is to keep the WCU_OFF_RAW frames, but have them optional. As in, the recipe should not fail if they are not there. We (PIP) then do not care whether these exposures are created or not.

Lastly it still fails on two tasks, namely the calibrate and post process task. This is since the telluric recipe right now only produces a FLUXCAL_TAB.fits, but IFU_TELLURIC.fits is needed. The rest should now execute.

We decided to change how the telluriccorrection is approached(AstarVienna/METIS_DRLD#310), so I suggest to just do whatever is necessary to make this work. E.g. either add the IFU_TELLURIC as output to metis_ifu_telluric, but simpler might be to just comment it out in the EDPS with a reference to AstarVienna/METIS_DRLD#310

But I do think we should only merge if the tests pass. (Or more specifically, if the workflow runs on the test data, which should be captured by the tests.)

[hugobuddel]

I managed to rebase main onto this and wrote a bunch of tests (mb/edps-tests). However, I got the same results, 8/15 working.

Small note, looking at the graph of the repository, it seems you indeed rebased main onto be/create_ifu_wkf. I don't think that is a logical approach. Normally one would either merge main into the feature branch (like I did on #90), or rebase the feature branch onto main. Rebasing main is only something for very special occasions I think, because the idea of rebasing is that one would discard the original branch (because rebasing duplicates the commits), and one would not normally discard main.

The above doesn't matter if mb/edps-tests is to be discarded, but I'm not sure that is the case. Several commits from main are now duplicated in mb/edps-tests due to the rebasing. Those duplicates should not be merged 'back' into main, since that needlessly pollutes the graph. But there are also commits in mb/edps-tests that are not anywhere else I think, so getting those cleanly in main requires yet another (interactive?) rebasing.

Also, if we want to add more EDPS tests, I suggest to directly do that in the ESO-way; see the HAWK-I example in https://github.com/AstarVienna/hawki-edps-tests

[sesquideus]

The above doesn't matter if mb/edps-tests is to be discarded, but I'm not sure that is the case. Several commits from main are now duplicated in mb/edps-tests due to the rebasing. Those duplicates should not be merged 'back' into main, since that needlessly pollutes the graph. But there are also commits in mb/edps-tests that are not anywhere else I think, so getting those cleanly in main requires yet another (interactive?) rebasing.

Sorry, yes, I failed to mention that mb/edps-tests was meant as a purely experimental branch. I just wanted to see if this worked and did not think of merging it back. And if there is a different way of automating EDPS tests, it's really immaterial.

[janusbrink]

Hi all - just a comment on the RSRF reipe and WCU_OFF_RAW frames. I had some discussions about this recipe during the METIS CM and the DRLD is out of date on a number of aspects that still needs TBC. The METIS Calibration Plan specifically calls for the WCU_OFF frames, for instance. For now I agree we can make these inputs optional in the skeleton recipe, but this will probably change going forward.

[eiseleb47]

I've tried merging main into the branch to get the updated Telluric Recipe from #94 . However it seems to be failing locally with this error: [WARNING] metis_ifu_telluric: No detector could be identified from the SOF

It passed the test in #94 because the run_edpy.yaml did only contain the LM workflow, thus not running the IFU Telluric task.

The test it ran now seems to also have passed the telluric recipe. I'm not quite sure why since I'm using a clean installation of the branch. Could you perhaps run it as well to see if it is an issue with something on my machine?

[sesquideus]

I am not sure what the problem might be. When I run this against METIS_Pipeline_Test_Data's main branch, I am not getting any errors. If you could show me your pyesorex.log and the SOF you're using, maybe I'll get some idea.

[eiseleb47]

That's weird. I'm using that branch as well. Do you mean the input.sof that is created by EDPS? I ran the recipe with Pyesorex directly using the input sof file from the metis_ifu_telluric recipe, but got the same error. They look like this:
sof file:

/home/kali/EDPS_data/METIS/metis_ifu_sci_reduce/043de482-142c-48cc-9c34-9fc78943e5cc/IFU_SCI_COMBINED.fits IFU_SCI_COMBINED
/home/kali/Documents/METIS_Pipeline_Test_Data/metis_sim_small_1/data/FLUXSTD_CATALOG.fits FLUXSTD_CATALOG
/home/kali/Documents/METIS_Pipeline_Test_Data/metis_sim_small_1/data/LSF_KERNEL.fits LSF_KERNEL
/home/kali/Documents/METIS_Pipeline_Test_Data/metis_sim_small_1/data/ATM_LINE_CAT.fits ATM_PROFILE

Error:

[ INFO  ] pyesorex: This is PyEsoRex, version 1.0.1.

     ***** ESO Recipe Execution Tool, Python version 1.0.1 *****

[ INFO  ] pyesorex: Loaded recipe 'metis_ifu_telluric'.
[ INFO  ] pyesorex: Running recipe 'metis_ifu_telluric'...
[WARNING] metis_ifu_telluric: No detector could be identified from the SOF
[ ERROR ] metis_ifu_telluric: Pyesorex raised cpl.core.FileIOError, exiting

[sesquideus]

By itself it should be fine, the warning should be removed anyway since this whole recipe is about IFU and detecting the detector does not make much sense. Could you please run something like this and show me?

pyesorex metis_ifu_telluric $SOF_DIR/metis_ifu_telluric.std.sof --log-level DEBUG

[hugobuddel]

The FileIOError might mean that you didn't set your paths right?

For me, metis_ifu_calibrate gives this error:

Start time     : 2025-02-06T21:08:44
Program name   : pyesorex
Severity level : [ INFO  ] 

21:08:44 [ INFO  ] load_recipe: [tid=000] Loaded recipe 'metis_ifu_calibrate'.
21:08:44 [ INFO  ] read_config: [tid=000] Read config file '/scratch/EDPS_data/METIS/metis_ifu_calibrate/0a6b9bba-4c07-4b54-be05-8d38f9f9f47e/parameters.rc'.
21:08:44 [ INFO  ] run: [tid=000] Running recipe 'metis_ifu_calibrate'...
21:08:44 [ ERROR ] MetisIfuCalibrateImpl: [tid=000] Data not found error: No flux calibration table frame (re.compile('FLUXCAL_TAB')) found in the frameset.
21:08:44 [ ERROR ] pyesorex: [tid=000] Pyesorex raised cpl.core.DataNotFoundError, exiting

The FLUXCAL_TAB is indeed not listed in the SOF file, but it does exist in ./metis_lm_img_standard_flux/d1d4821f-83aa-4a8e-9a17-d52d83030b6a/FLUXCAL_TAB.fits

metis_ifu_postprocess also fails with an edps error because metis_ifu_calibrate fails:

$ cat error.log 
e5cf61c8-9ef8-4cf3-9fe7-110636a93a2b metis_ifu_postprocess e5cf61c8-9ef8-4cf3-9fe7-110636a93a2b ([UUID('0a6b9bba-4c07-4b54-be05-8d38f9f9f47e')]) ([]) job execution crashed prematurely because: Parent 0a6b9bba-4c07-4b54-be05-8d38f9f9f47e metis_ifu_calibrate 0a6b9bba-4c07-4b54-be05-8d38f9f9f47e ([UUID('19457e0b-33ba-44bf-8a8a-8f04ae822735')]) ([UUID('02690492-c129-47ae-a731-21b40ac6efa4'), UUID('3380c8bc-c72f-446e-9237-3c12ea25d731')]) execution failed

[eiseleb47]

@sesquideus The output looks like this for me:

(Metis_Pip) kali@DESKTOP-LIK3UGF:~/Documents/METIS_Pipeline$ pyesorex metis_ifu_telluric $SOF_DIR/metis_ifu_telluric.std.sof --log-level DEBUG
[ INFO  ] pyesorex: This is PyEsoRex, version 1.0.1.

     ***** ESO Recipe Execution Tool, Python version 1.0.1 *****

[ INFO  ] pyesorex: Loaded recipe 'metis_ifu_telluric'.
[ ERROR ] pyesorex: Pyesorex raised cpl.core.FileIOError, exiting

When trying to execute it via the python command line I got this result when I tried to select the sof file:

>>> p.sof_location="/home/kali/Documents/METIS_Pipeline_Test_Data/metis_sim_small_1/sof/metis_ifu_telluric.std.sof"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kali/.environments/Metis_Pip/lib/python3.11/site-packages/pyesorex/pyesorex.py", line 579, in sof_location
    self._sof = ui.FrameSet(self._sof_location)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cpl.core.FileIOError: Most recent error last:
  File "/tmp/pip-install-v8d5ixsn/pycpl_0e06da0083c846a0bc2512cc76087099/src/cplui/frame.cpp", line 58, in Frame
/IFU_STD_COMBINED.fits could not be found

@hugobuddel perhaps it does have to do with the paths. Currently I have them set up like this:

export PYESOREX_PLUGING_DIR="/home/kali/Documents/METIS_Pipeline/metisp/pyrecipes/"
export PYCPL_RECIPE_DIR="/home/kali/Documents/METIS_Pipeline/metisp/pyrecipes/"
export METIS_SOFTPATH="/home/kali/Documents/METIS_Pipeline/metisp/pymetis/src"
export PYTHONPATH=$METIS_SOFTPATH
export SOF_DATA="/home/kali/Documents/METIS_Pipeline_Test_Data/metis_sim_small_1/data"

However they do seem to be the same as in the run_edps.yaml file.

[sesquideus]

@eiseleb47 Ah yes, I see now.

You can get the full output with --log-level DEBUG, but only to pyesorex.log, not stdout. Then these things usually boil down to hunting for the file and trying to find out whether it is actually missing, or your envvars pointing to the wrong place, or the tag regex being incorrect, or whatever.

[JenniferKarr]

There are some issues with the sofFiles, $SOF_DATA and where the intermediate files end up. I generated the release sof files with the assumption that all the files would be in the same location, $SOF_DATA, but if you run a recipe at the pyesorex level, it writes the output to the directory in which you run the command, which means it can't find files generated in an earlier step.

I think we could solve that by calling pyesorex --output-dir $SOF_DATA which would keep all the data in one place.

[sesquideus]

I set it directly in my shell rc:

export PYESOREX_OUTPUT_DIR=$SOF_DATA

though I don't know if this solves everything -- it does locally, in the sense that eventually all tests run if repeated long enough.

[hugobuddel]

On the telecon this morning we determined that the missing FLUXCAL_TAB is due to metis_ifu_telluric creating two IFU_TELLURIC products, instead of one IFU_TELLURIC and one FLUXCAL_TAB; that is, this line is wrong:

METIS_Pipeline/metisp/pymetis/src/pymetis/recipes/ifu/metis_ifu_telluric.py

Line 87 in c814346

tag = r"IFU_TELLURIC"

@janusbrink offered to fix it

[hugobuddel]

The PYESOREX_OUTPUT_DIR is also set in the github action, so adding the pyesorex runs in the correct order works, see

METIS_Pipeline/.github/workflows/run_edps.yaml

Line 53 in c814346

export PYESOREX_OUTPUT_DIR="$SOF_DATA"

But of course it only works if the recipes actually produce the data that they should, see comment above.

[eiseleb47]

I set the variable for PYESOREX_OUTPUT_DIR and implemented the changes to metis_ifu_telluric from baeb560.
It works locally and also also passes the tests on this branch.

I've held of on merging the rest of main into this branch to make sure that it would work since the tests for the newest commit don't pass as of now.

Additionally, right now there is no implementation of the optional input for the callibration task like it is shown in the IFU flowchart in the DRLD. Thus I am not sure if it takes the IFU Telluric outputs from the SCI or STD task. However I'd say, since it is still a skeleton, this would be something for the next realease.

[hugobuddel]

Everything does seem to work:

00h00m05s {'COMPLETED': 23} 
23 jobs processed in 00h00m05s
You can find reduced data, logs and quality control plots in the following directories:
=======================================================================================
/tmp/EDPS_data/METIS/metis_ifu_lingain/88314499-06b6-49f0-9b2b-0eb5e6c91d7e
/tmp/EDPS_data/METIS/metis_ifu_dark/3877b49b-4999-49b6-b202-8213f60f1b64
/tmp/EDPS_data/METIS/metis_ifu_distortion/6a09a14e-cf19-4ec3-b336-3293b3836bcf
/tmp/EDPS_data/METIS/metis_ifu_wavecal/a7b44378-1b2c-4468-9713-3b975ef181eb
/tmp/EDPS_data/METIS/metis_ifu_rsrf/85915550-553e-4dba-90ea-202af2220360
/tmp/EDPS_data/METIS/metis_ifu_std_reduce/f0891cc5-a446-4c9f-bc8d-0fae837c91d2
/tmp/EDPS_data/METIS/metis_ifu_std_telluric/aee6b475-c76d-4b43-8bf4-394b4927e7f9
/tmp/EDPS_data/METIS/metis_ifu_sci_reduce/d79704b2-5dee-415f-92b6-8b178e69e1d5
/tmp/EDPS_data/METIS/metis_ifu_sci_telluric/31c02d8e-c73b-4982-aa0d-028e5fd12ee5
/tmp/EDPS_data/METIS/metis_ifu_calibrate/e0832b92-71c8-4664-ba2c-1354ea80cedd
/tmp/EDPS_data/METIS/metis_ifu_postprocess/70734119-45cf-4a66-bb2f-5dc0e20736e3
/tmp/EDPS_data/METIS/metis_det_dark/da0e776a-4377-4190-8852-1486571cb3fe
/tmp/EDPS_data/METIS/metis_det_detlin/0c60d122-c1c6-44a0-b56d-80c0ff7edffa
/tmp/EDPS_data/METIS/metis_lm_img_cal_distortion/607d3f07-cae3-42cf-a99d-68bf081fad86
/tmp/EDPS_data/METIS/metis_lm_img_flat/5a0ee546-d007-4baa-82cc-2e18bd8500c7
/tmp/EDPS_data/METIS/metis_lm_img_basic_reduce_sky/f71b8904-5490-4f24-8c77-c1fd5d9b858e
/tmp/EDPS_data/METIS/metis_lm_img_basic_reduce_std/d3332c90-1a24-4bc9-b9b7-372f695fee93
/tmp/EDPS_data/METIS/metis_lm_img_background_std/723f1908-73f3-4c4a-99bc-aa65b5eb43ca
/tmp/EDPS_data/METIS/metis_lm_img_standard_flux/d95302a4-9e86-4245-803f-f61da3a5c79c
/tmp/EDPS_data/METIS/metis_lm_img_basic_reduce_sci/69036d74-af33-46e3-9d20-a765e9152ac6
/tmp/EDPS_data/METIS/metis_lm_img_background_sci/3dccfde5-11fd-4119-aa07-a320a12d4f29
/tmp/EDPS_data/METIS/metis_lm_img_calib/c8df9377-994b-490d-b98d-9f1dce1ef6af
/tmp/EDPS_data/METIS/metis_lm_img_coadd/3f7b18ad-ba51-4121-9987-02e589ee4e49

[sesquideus]

Works for me too.

[sesquideus]

I was able to run everything and got COMPLETED.