Archives tools prototypes #25

francoismg · 2023-08-02T14:36:18Z

Tools prototypes for archives browsing

tools/archives/pyvo_integration/astronomical_archives.xml

volodymyrss · 2023-08-03T08:39:17Z

tools/archives/esap_integration/astronomical_archives.xml

volodymyrss · 2023-08-11T09:20:44Z

@francoismg I put some checkboxes. Could you see that it makes sense? Please ask questions if unclear or there are any roadblocks!

volodymyrss · 2023-08-11T09:31:43Z

just to the record, we might end up using an interactive tool in the future, e.g. like #26

francoismg · 2023-08-11T09:35:38Z

@francoismg I put some checkboxes. Could you see that it makes sense? Please ask questions if unclear or there are any roadblocks!

Yes everything seem clear right now, I'm just not sure about what you want to add in the citations section.

Also the for the tap explorer I'm not sure that we can display something like this directly from a static file but I will try it and let you know.

francoismg · 2023-08-11T09:37:12Z

just to the record, we might end up using an interactive tool in the future, e.g. like #26

yea sure no problem, I thought it was already decided that it would probably be an interactive tool

volodymyrss · 2023-08-11T09:41:53Z

just to the record, we might end up using an interactive tool in the future, e.g. like #26

yea sure no problem, I thought it was already decided that it would probably be an interactive tool

We might be talking about different things. Let's discuss later. But for now, let's just finish this first basic but already useful tool and find out with upstream developers how to best fill the gaps.

volodymyrss · 2023-08-11T09:46:03Z

@francoismg I put some checkboxes. Could you see that it makes sense? Please ask questions if unclear or there are any roadblocks!

Yes everything seem clear right now, I'm just not sure about what you want to add in the citations section.

we want several things here, specifically:

10.3847/1538-3881/aabc4f

etc from

https://github.com/astropy/astropy/blob/main/astropy/CITATION

for https://github.com/astropy/pyvo I do not see a DOI directly, just put the URL then

for https://www.ivoa.net/documents/latest/ADQL.html

10.5479/ADS/bib/2008ivoa.spec.1030O

please put these and also a TODO to let me think more what to add

Also the for the tap explorer I'm not sure that we can display something like this directly from a static file but I will try it and let you know.

Actually as discussed let's postpone this tap explored until after delivery of the first tool version. I will indicate this in the list.

tools/archives/pyvo_integration/astronomical_archives.xml

…rameter order + changed file output selection + better error handling and reporting + added citations + changed test cases + added basic output html as output choice

volodymyrss · 2023-08-17T12:08:03Z

tools/archives/pyvo_integration/astronomical_archives.xml

+              <param name="t_min" type="text" label="Start time in MJD" />
+              <param name="t_max" type="text" label="Stop time in MJD" />
+              <param name="dataproduct_type" type="text" label="Data product type" help="Logical data product type (image etc.)" />
+              <param name="dataproduct_type_select" type="select" label="Data product type" help="Logical data product type">


please add reference where this comes from

tools/archives/pyvo_integration/astronomical_archives.xml

volodymyrss · 2023-08-17T12:46:49Z

tools/archives/pyvo_integration/astronomical_archives.py

+            try:
+                fits_file = FileHandler.download_file_from_url(url[access_url])
+                FileHandler.write_file_to_subdir(fits_file, FileHandler.get_file_name_from_url(url[access_url]))
+            except Exception as e:


handle this exception with some extra output to the user

volodymyrss · 2023-08-17T12:47:02Z

tools/archives/pyvo_integration/astronomical_archives.py

+            access_url = 'access_url'
+
+        fits_file = FileHandler.download_file_from_url(file_url[0][access_url])
+        FileHandler.write_file_to_output(fits_file, output, "wb")


portentially this fails?

volodymyrss · 2023-08-17T12:48:22Z

tools/archives/pyvo_integration/astronomical_archives.xml

+        </data>
+        <data name="output_error" format="txt" label="${tool.name} -> Error Log:"></data>
+    </outputs>
+    <tests>


add tests with failure, empty archive, no obscore, etc

tools/archives/pyvo_integration/astronomical_archives.xml

volodymyrss · 2023-08-17T13:42:58Z

tools/archives/pyvo_integration/astronomical_archives.xml

@@ -0,0 +1,1005 @@
+<tool id="astronomical_archives" name="Astronomical archives" version="1.0.0">
+    <description>queries astronomical archives</description>


Suggested change

<description>queries astronomical archives</description>

<description>queries astronomical archives through Virtual Observatory protocols</description>

volodymyrss · 2023-08-17T13:49:35Z

tools/archives/pyvo_integration/astronomical_archives.xml

+            </param>
+            <when value="registry">
+              <param name="keyword" type="text" label="Keyword" />
+              <param name="service_type" type="select" label="Service type">


please put a reference for versionned service type vocabulary. also for all other select options which rely on a particular version of some vocabularies

tools/archives/pyvo_integration/astronomical_archives.xml

bgruening · 2023-08-30T12:34:00Z

@francoismg @volodymyrss should I review again? When is your talk, we need to get this installed by then, do we?

volodymyrss · 2023-08-30T12:51:15Z

@francoismg @volodymyrss should I review again? When is your talk, we need to get this installed by then, do we?

Yes, it would be good. It's next Wednesday.
I made a couple of changes but we need @francoismg to finish what was being done.

volodymyrss · 2023-08-30T12:52:13Z

When should we aim to install it so that it can appear on time? Assuming it might not workout at once.

bgruening · 2023-08-30T12:55:52Z

Once its merged I can install it manually. So this can be fast. But I would say Monday latest?

francoismg · 2023-08-30T13:15:47Z

@francoismg @volodymyrss should I review again? When is your talk, we need to get this installed by then, do we?

Yes, it would be good. It's next Wednesday. I made a couple of changes but we need @francoismg to finish what was being done.

I reworked how the tool run now that it get the outputs from json, it seems more readable and less clutered that way but took me longer than I thought to fix all the bugs. RIght now everything seems to be working and I have merged the changes from that PR. I am merging the changes from your other PR right now so should be good soon

…es and clean up

bgruening · 2023-08-31T05:39:51Z

tools/archives/pyvo_integration/astronomical_archives.xml

+            </when>
+            <when value="archive">
+              <param name="archive" type="select" label="Astronomical archives">
+                <options from_data_table="astronomical_archives_1" />


why has this a _1 ?

my bad thanks, forgot to change after moving the code

bgruening · 2023-08-31T05:45:56Z

Linting is failing because of Python flake8 errors.

bgruening · 2023-08-31T05:48:22Z

tools/archives/pyvo_integration/astronomical_archives.xml

+            <param name="output_selection" value="i"/>
+            <param name="number_of_files" value="1"/>
+            <conditional name="archive_selection">
+                <param name="archive_type" value="registry"/>


it would be nice to have at least one test using the data_table.

I had some when I was using "from file" instead of a data_table but when running planemo test I always got some "invalid value" or something like that.

I could try it again now that we moved to data_table

maybe that one helps you https://github.com/galaxyproject/tools-iuc/tree/96f8a533278b4b6394aebd7a8f537513b0d29b1a/tools/recentrifuge

bgruening · 2023-08-31T05:49:50Z

tools/archives/pyvo_integration/astronomical_archives.xml

+            </assert_contents>
+          </output>
+        </test>
+        <test expect_num_outputs="1">


Mh, the tool does not fail when we have an error in the query?

Would that not be the expected behaviour from a users point of view.
You can test for error code and such. But this needs to be implemented in your python script I guess, that you really fail.

when a query fail we update the summary output to inform the user so he can possibly fix the error in the query, if the tool failed the user would not get the summary output right or am I mistaken?

Yes, if it fails the outputs is not returned. But you get the stdout/stderr.

ok I thought it would be more user friendly if we informed the user through some summary file but we could just output everything in the stdout if you think it would be better no problem

Your call, I don't know :)
I'm happy to +1 it as it is :)

bgruening · 2023-09-04T09:56:25Z

If you merge please create a pr to the tools repo I will make sure it gets installed during your night latest.
Thanks this is awesome!

volodymyrss · 2023-09-04T10:14:11Z

@francoismg you want to add it? Here is an example.

francoismg · 2023-09-04T10:18:34Z

@francoismg you want to add it? Here is an example.

yea sure will try that

francoismg · 2023-09-04T10:53:36Z

@francoismg you want to add it? Here is an example.

So just to be sure I understand correctly.

I should make a pr to the usegalaxy-eu-tools repo to update the astronomy yaml with the new tool reference and after that push the actual tool files to another repo is that correct?

You said you pushed your tools with planemo but that it caused some issues right? So after updating the yaml file I should make a pr to @bgruening repo https://github.com/bgruening/galaxytools or the IUC repo https://github.com/galaxyproject/tools-iuc with the actual tool files, does that make sense?

volodymyrss · 2023-09-04T11:56:45Z

@francoismg you want to add it? Here is an example.

So just to be sure I understand correctly.

I should make a pr to the usegalaxy-eu-tools repo to update the astronomy yaml with the new tool reference and after that push the actual tool files to another repo is that correct?

You said you pushed your tools with planemo but that it caused some issues right? So after updating the yaml file I should make a pr to @bgruening repo https://github.com/bgruening/galaxytools or the IUC repo https://github.com/galaxyproject/tools-iuc with the actual tool files, does that make sense?

Not quite.

This tools-astro repository here has CI/CD which pushes to the toolsched. It just failed due to, apparently, toolsched being down?.. It seems back now very slow to respond, let's see if CI completes.

Once pushed to toolshed, the tools are added to the galaxy EU instance with esg-epfl-apc/usegalaxy-eu-tools#1

francoismg · 2023-09-04T12:01:15Z

ok thanks, so just need to wait then

francoismg · 2023-09-04T12:04:22Z

Repository [suite_astronomicalarchivestools] does not exist in the targeted Tool Shed.
Failed to update repository 'suite_astronomicalarchivestools' as it does not exist on the main Tool Shed.

There's also that error, I followed the other yaml file you made and created another suite but apparently that might not be correct

volodymyrss · 2023-09-04T12:35:33Z

The problem you quote was with access and I suspect only that, all the following problems follow it.

The problem now is this one:

<br/><b>astronomical_archives.xml<\/b> 
- This file refers to a file named <b>astronomical_archives_gen.loc<\/b>.  
- Upload a file named <b>astronomical_archives_gen.loc.sample<\/b> to the repository to correct this error.<br/>"}

francoismg · 2023-09-04T12:45:45Z

The problem you quote was with access and I suspect only that, all the following problems follow it.

The problem now is this one:
 astronomical_archives.xml<\/b> 
- This file refers to a file named astronomical_archives_gen.loc<\/b>. 
- Upload a file named astronomical_archives_gen.loc.sample<\/b> to the repository to correct this error. "}

I pushed a commit with a loc.sample file but that doesn't seem to work since the pr has been merged (i think), will open another pr

francoismg added 3 commits July 17, 2023 08:37

archives tool prototypes

da5f87a

archives tool prototypes

4d7e6d2

archives tool prototypes

f00b5d1

volodymyrss reviewed Aug 2, 2023

View reviewed changes

tools/archives/pyvo_integration/astronomical_archives.xml Outdated Show resolved Hide resolved

archives tool prototypes

75e56da

volodymyrss reviewed Aug 2, 2023

View reviewed changes

tools/archives/pyvo_integration/astronomical_archives.xml Outdated Show resolved Hide resolved

volodymyrss reviewed Aug 2, 2023

View reviewed changes

tools/archives/pyvo_integration/astronomical_archives.xml Outdated Show resolved Hide resolved

html preview + tests + help + various fixes

547e292

volodymyrss requested a review from dsavchenko August 10, 2023 13:16

volodymyrss reviewed Aug 11, 2023

View reviewed changes

tools/archives/esap_integration/astronomical_archives.xml Outdated Show resolved Hide resolved

volodymyrss reviewed Aug 11, 2023

View reviewed changes

tools/archives/pyvo_integration/astronomical_archives.xml Show resolved Hide resolved

removed esap integration from pr + changed section order + changed pa…

781f1fb

…rameter order + changed file output selection + better error handling and reporting + added citations + changed test cases + added basic output html as output choice

volodymyrss reviewed Aug 17, 2023

View reviewed changes