Skip to content

Commit

Permalink
Fixes from review
Browse files Browse the repository at this point in the history
  • Loading branch information
timj committed Dec 4, 2024
1 parent 0848e96 commit a666d88
Showing 1 changed file with 12 additions and 12 deletions.
24 changes: 12 additions & 12 deletions P920.tex
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
\begin{abstract}
The IVOA Simple Image Access version 2 protocol defines an easy way to provide community access to a collection of data.
At the Vera C.\ Rubin Observatory we currently enable ObsTAP access to our data holdings via an ObsCore export or view of our Data Butler repositories.
This approach does come with some deployment constraints, such as requiring pgsphere and compatibility with our CADC TAP implementation, so recently we decided to see whether we could instead provide an SIAv2 service that talks directly to our Data Butler.
This approach comes with some deployment constraints, such as requiring pgsphere and compatibility with our CADC TAP implementation, so recently we decided to see whether we could instead provide an SIAv2 service that talks directly to our Data Butler.
Here we describe our motivation, implementation strategies, and current deployment status, as well as discussing some metadata mismatches between the Butler data models and SIAv2.
\end{abstract}

Expand All @@ -66,12 +66,12 @@ \section{Motivation}
Both these approaches have been implemented by us and are the only way we had to provide an ObsTAP service \citep{2017ivoa.spec.0509L}.
However, there are downsides.
Whilst static export works fine for formal data releases where it can be integrated into our high performance Qserv database \citep{C15_adassxxxii}, it is not suitable for evolving datasets such as those from the nightly prompt products.
``Live'' ObsCore does work but requires that the deployment has pgsphere available.
Also, if the configuration changes (something that can happen a lot during early operations) the entire table needs to be rebuilt.
``Live'' ObsCore works but requires that the deployment has pgsphere available.
Also, if the configuration changes (something that can happen frequently during early operations) the entire table needs to be rebuilt.

These two issues suggested that there would be a benefit from providing a simpler, yet standardized, query layer directly on top of the Butler.
These two issues suggested that there would be a benefit from providing a simpler, yet standardized, query layer directly atop the Butler.
The IVOA's Simple Image Access version 2 \citep[SIAv2;][]{2015ivoa.spec.1223D} was the obvious choice for this.
Interfacing directly to the Butler provides much more flexibility since configuration changes can be picked up with a simple redeployment of the service and it can work immediately with any Butler repository.
Interfacing directly to the Butler provides much more flexibility since configuration changes can be picked up with a simple restart of the service and it can work immediately with any Butler repository.

\section{Implementation}

Expand Down Expand Up @@ -108,11 +108,11 @@ \section{Current Status}
and \texttt{COLLECTION} are coming soon.
For the \texttt{FACILITY} keyword we intend to match the AAS Facility names of ``Rubin:Simonyi'' for the Simonyi Survey Telescope and ``Rubin:1.2m'' for the Rubin Auxiliary Telescope.
The Butler does not have a concept of facility but the correct value will be derived from the instrument name.
Butler does have a concept of target but in general the value of the field is not of interest since it will usually refer to a field name generated by the survey scheduler.
Butler does have a concept of target, but in general, the value of the field is not of interest, since it will usually refer to a field name generated by the survey scheduler.

\section{Model Mismatches}

The Butler data model as used by Rubin Observatory does result in some implementation difficulties.
The Butler data model as used by Rubin Observatory results in some implementation difficulties.

\subsection{Instrument for coadds}

Expand All @@ -124,28 +124,28 @@ \subsection{Instrument for coadds}

\subsection{Exposure time for co-adds}

Individual observations do include exposure times in the Butler registry metadata but this information is not available to co-adds.
This is because the median exposure time for a co-add is a derived quantity that is not know when the Butler coordinate space is defined.
Individual observations include exposure times in the Butler registry metadata, but this information is not available to co-adds.
This is because the median exposure time for a co-add is a derived quantity that is not known when the Butler coordinate space is defined.
Storing derived metadata, which is similar to calculating the seeing and then allowing a query based on seeing, is on the future development roadmap.

\subsection{Observing dates for co-adds}

Butler does know the observing dates of individual observations but for a co-add this information is lost.
Butler knows the observing dates of individual observations, but for a co-add this information is lost.
In the future when a full Butler provenance system is implemented it may be possible to derive the date range for co-adds, but it is not possible at this time.

\subsection{Dataset Types}

Butler makes extensive use of what we call ``dataset types'' which define each product type in a pipeline.
Examples can be \texttt{visit\_image}, \texttt{difference\_image}, and \texttt{deep\_coadd}.
Currently there is no standardized way in SIAv2 for a query to specify these.
We are looking at adding an extension, potentially using a \texttt{DPSUBTYPE} query parameter that will map directly to the Butler dataset type, possibly with an `lsst` prefix.
We are looking at adding an extension, potentially using a \texttt{DPSUBTYPE} query parameter that will map directly to the Butler dataset type, possibly with an \texttt{lsst} prefix.

\section{Conclusions}

Implementing SIAv2 over the Data Butler was a relatively simple process.
Separating the development into a standalone SIAv2 parameters handler in \texttt{dax\_obscore} with a service that forwards parameters to this layer allowed for parallel development.
An additional advantage is that there is now a command-line tool that can be used to perform SIAv2-style queries directly on any Butler repository.
Some data model mismatches are causing difficulties but we hope to fix most of those issues with the additional support for querying on derived metadata and provenance.
Some data model mismatches are causing difficulties, but we hope to fix most of those issues with the additional support for querying on derived metadata and provenance.

\acknowledgments This material or work is supported in part by the National Science Foundation through Cooperative Agreement AST-1258333 and Cooperative Support Agreement AST1836783 managed by the Association of Universities for Research in Astronomy (AURA), and the Department of Energy under Contract No.\ DE-AC02-76SF00515 with the SLAC National Accelerator Laboratory managed by Stanford University.

Expand Down

0 comments on commit a666d88

Please sign in to comment.