Publish to Julia Registry?

[tamasgal]

I think the only missing thing is a little bit of documentation and this awesome package could be published. Or are there any objections?

[barrettp]

This package isn't complete. The main Astrometry package still needs to be fleshed out, particularly the timescales types need to be implemented. The SOFA sub-package still needs about a dozen functions implemented and the documentation reformatted. The SOFA sub-package is primarily a support package written to check the accuracy of the main Astrometry package. That and Dave MacMahon requested a drop-in Julia version of the SOFA library. I'd prefer not to add it the Julia Registry at this time, because I don't want to give users the impression that the SOFA sub-package is the primary package. I'd like users to use the smaller, yet more user friendly, main package.

[tamasgal]

To me it looks like the SOFA subpackage of this package is more of an internal thing, so if you don't export it, people will not use it. Or maybe I don't see the full picture 😉 You mean that people might confuse it with https://github.com/sisl/SOFA.jl ?

Do I understand correctly that the SOFA subpackage is meant to be removed in future or do you want to leave it in place for crosschecks in the test suite? I think if that package is hidden from the public API, there is no reason to worry that people will use it as a primary package. On the other hand, it would be nice to have a full Julia implementation of the original SOFA routines as well.

[barrettp]

The SOFA sub-package is meant to be a migration path for those who are already using the C interface package, i.e., the sisl package. It makes for a drop-in replacement. It also has the potential for checking the accuracy of the C algorithms by replacing the Float64 type with higher precision types such as Float128 or BigFloat and for checking the error propagation of the algorithms. These issues become more important as astrometry and timescales becomes more accurate. My guess is that the accuracy of the C/FORTRAN algorithms are overstated because no one has done an error propagation analysis of the algorithms. Julia has the potential to make this analysis relatively easy by redefining the input types.

Should the SOFA sub-package be decoupled from the main Astrometry package? Probably. And it may be done in the future.

[barrettp]

The SOFA sub-package is also meant to provide a potentially higher performance version of the C library, particularly when doing calculations of a large number (of order millions) of objects. Having to do a C call on every object incurs a huge overhead and is a really big performance hit.

[tamasgal]

OK I see, thanks for the explanation.

It's still not fully clear to me what the actual path is to make Astrometry a "real" package. I think it would be good if there was a consolidation of missing features which are required to make it official ;) I can try to help out but not sure how... For me, this package is already useful as it is (already have a few thousand jobs in the production pipeline) but it would clearly be better if I had a version to refer to. Currently I am using my own fork which has a tagged version for the sake of reproducibility.

[barrettp]

The Astrometry package needs well implemented timescales and reference frames types that allows implicit conversions between the various timescales and reference frames. So, for example, when a function's argument signature requires a UTC date in days, the input type is automatically converted to the correct timescale and unit. This reduces human error and ensures the results are correct. I'm not entirely happy with current AstroJulia timescales package because it combines units and timescales into a single type when they should be orthogonal types. I've not had time to work on this and also want to finish the SOFA implementation first, hence, the delay.

The precession-nutation function in the main Astrometry package gives a good example of the implementation and interface. It combines all the precession nutation algorithms into a single function with the default being the 2006A algorithm. This simplifies the interface by defaulting to the most commonly requested algorithm. I hope that clarifies the issue a bit more.

[tamasgal]

Absolutely, I fully understand. If there is anything I can do, let me know. A todo list would probably be a good idea to be able to pick low-hanging fruits ;)

[barrettp]

The one task that would be helpful is to ensure that all of the doc strings for each SOFA function are presented consistently and correctly, i.e., they begin with the function signature, followed by the Input and Output values, and then Note and References sections, if applicable. Many of the doc strings still refer to the C function names. It would be nice, although not necessary, to have them refer to the Julia function names. It would also be nice to replace the Greek words with their corresponding characters. In summary, it would be nice to have good doc strings.

I'm slowly working to complete the remaining dozen or so star catalog functions, e.g., fk425, and their tests. Once this is done, I will tag the package version as 0.2.

[MicheleCeresoli]

The Astrometry package needs well implemented timescales and reference frames types that allows implicit conversions between the various timescales and reference frames. So, for example, when a function's argument signature requires a UTC date in days, the input type is automatically converted to the correct timescale and unit. This reduces human error and ensures the results are correct. I'm not entirely happy with current AstroJulia timescales package because it combines units and timescales into a single type when they should be orthogonal types. I've not had time to work on this and also want to finish the SOFA implementation first, hence, the delay.

The precession-nutation function in the main Astrometry package gives a good example of the implementation and interface. It combines all the precession nutation algorithms into a single function with the default being the 2006A algorithm. This simplifies the interface by defaulting to the most commonly requested algorithm. I hope that clarifies the issue a bit more.

Hi @barrettp, I happened to have a look at the slack Astronomy channel and found the link to this repository!
Its nice to see the great deal of work you've done to translate the SOFA routines into pure Julia code and I completely agree on what you said about the benefits that this could bring.

Me and @andreapasquale94 had already a Julia implementation of most SOFA routines (for both time and earth frame rotations) within FrameTransformations. With its next version we were planning to move all the IERS-related functionalities into a completely standalone package (IERS.jl), which at the current iteration is capable of supporting all the routines associated with both the equinox-based and CIO-based approaches, with native support for all the following rotations:

• Equinox-based: GCRF\ITRF to\from MOD, TOD, GTOD and PEF.
• CIO-based: GCRF\ITRF to CIRF and TIRF.

All of its functionalities also support a variety of models: all of the IERS 1996/2003/2010 conventions, as well as two simplified models from Capitaine. At the current state the package is almost ready to be released (we are just missing a small rework of the functions that handle EOP data and some documentation\tests).

Since within the Julia ecosystem there are already some packages that support some of these routines (i.e., EarthOrientation.jl and SatelliteToolboxTransformations.jl, I was thinking that maybe we could help each other out and leverage the functionalities exported by our package for all the back-end work, while Astronometry.jl could add on top of that the more astronomy-related functionaliities: in the next releases we are planning to implement support for the missing IERS functionalities (e..g, solid tides as well as diurnal and semidiurnal variations produced by ocean tides) but won't be adding much more than that.

What do you think about this?

PS: we also have developed our own package to support time transformations between different timescales, which at the current iteration is quite similar to AstroTime, but I would also like to hear more about what you think are the needs in this regard. In our framework the IERS package is meant to be used as low-level dependency by other packages, and for that reason all of its routines require the time argument to be expressed in Terrestrial Time Julian centuries or seconds since J2000 (except the ERA functions which require UT1 days)

[barrettp]

@MicheleCeresoli Yes, cooperation and coordination among various packages would be most helpful and beneficial.

One of the main reasons for implementing the SOFA/ERFA routines in Julia is performance. Wrapping the C versions of these libraries is fine, if one only needs to perform a calculation on a few objects. However, if one is doing this on a catalog of millions, or possibly billions, of objects, then one begins to see the lack of performance of doing each calculation individually, which is essentially what the C version is doing. I experienced this situation 15 years ago when we created the Python wrapper for the C version of the Naval Observatory Vector Astrometry Software (NOVAS) library. It took at least 10 minutes to precess and nutate over one million stars, which is unacceptable in my opinion. So, the eventual goal of having a Julia implementation of these libraries is to be able to vectorize this calculation for millions of stars to get the best performance possible and hopefully to reduce the calculation to seconds instead of minutes.

A secondary goal is to test and to improve these algorithms by calculating them at higher precision, such as Float128, and investigating the effect of propagating the errors associated with the constants. I doubt this latter calculation has been done by the SOFA board. If you have looked at the implementation of these algorithms, you'll see that they are not well implemented. The coding style is based on FORTRAN from 40 years ago. They are not implemented with performance in mind.

Note that the reasons for implementing the SOFA/ERFA functions is to provide (1) a drop-in replacement for the C versions (as requested by Dave MacMahon), and more importantly, (2) a check on a rewrite of the algorithms using the array and matrix features of Julia.

As noted previously, the timescales package needs to separate the various timescales, units, and reference frames. In addition, I would like to see the timescales package accommodate precisions of order attoseconds or better. TAI is currently at a precision of order tens of femtoseconds. With the advent of optical and nuclear clocks, the precision of TAI will increase to that of attoseconds and eventually zeptoseconds. I think the timescales package should be prepared to accommodate such precisions. The timescale package is not solely for supporting the Astrometry package. It is a critical, independent package of its own. I think such preparation will result in a much better package and hopefully, dispel any doubt about the accuracy of the timescale package.

I think the next step would be to create a dependency graph of the various packages to see how they relate.

[MicheleCeresoli]

One of the main reasons for implementing the SOFA/ERFA routines in Julia is performance. Wrapping the C versions of these libraries is fine, if one only needs to perform a calculation on a few objects. However, if one is doing this on a catalog of millions, or possibly billions, of objects, then one begins to see the lack of performance of doing each calculation individually, which is essentially what the C version is doing. I experienced this situation 15 years ago when we created the Python wrapper for the C version of the Naval Observatory Vector Astrometry Software (NOVAS) library. It took at least 10 minutes to precess and nutate over one million stars, which is unacceptable in my opinion. So, the eventual goal of having a Julia implementation of these libraries is to be able to vectorize this calculation for millions of stars to get the best performance possible and hopefully to reduce the calculation to seconds instead of minutes.

Yes I agree on the need to improve the execution speed of those routines (mainly the lengthy computations of the nutation components and\or the CIP coordinates). Indeed, in our package we have already implemented an automatic parser of all the constants that minimises the number of operations (e.g., removes all multiplications with null coefficients, minimises the number of repeated sin\cos with the same argument, etc...). Doing so has already more than halved the execution speed with respect to the original C implementation (it takes about 35 microseconds on a basic laptop to compute the full ITRF-to-GCRF matrix with the IAU2006/2000A model, which would bring the time to about 40 seconds for one million stars). I'm a bit skeptical about vectorising all those functions since it could lead to some serious performance drawbacks if we don't prevent memory allocations.

A secondary goal is to test and to improve these algorithms by calculating them at higher precision, such as Float128, and investigating the effect of propagating the errors associated with the constants. I doubt this latter calculation has been done by the SOFA board. If you have looked at the implementation of these algorithms, you'll see that they are not well implemented. The coding style is based on FORTRAN from 40 years ago. They are not implemented with performance in mind.

This should be quite easy since we already have a Julia version of those algorithms: simply replacing the time arguments (or even how the constants are initially stored for investigation purposes) already allows to retrieve the outputs with the desired precision. I don't know which routines you are referring to, but most of those associated with precession\nutation simply consists in basic polynomial evaluations which already implement Horner's method, so I don't know how much execution speed can be further improved there.

Note that the reasons for implementing the SOFA/ERFA functions is to provide (1) a drop-in replacement for the C versions (as requested by Dave MacMahon), and more importantly, (2) a check on a rewrite of the algorithms using the array and matrix features of Julia.

Apart from speed, our needs for having a pure Julia implementation were also associated to the possibiliy of using Automatic Differentiation algorithms across all of these calls. I'm not totally sure whether point (1) requires those routines to be exactly equal to those of SOFA (for comparison purposes) or just requires them in pure Julia. In any case, we've already got most of that covered with errors with respect to ERFA below 1 microarcseconds (i.e., below the actual IAU2006/2000A model precision).

As noted previously, the timescales package needs to separate the various timescales, units, and reference frames. In addition, I would like to see the timescales package accommodate precisions of order attoseconds or better. TAI is currently at a precision of order tens of femtoseconds. With the advent of optical and nuclear clocks, the precision of TAI will increase to that of attoseconds and eventually zeptoseconds. I think the timescales package should be prepared to accommodate such precisions. The timescale package is not solely for supporting the Astrometry package. It is a critical, independent package of its own. I think such preparation will result in a much better package and hopefully, dispel any doubt about the accuracy of the timescale package.

I agree the timescale handling package should be a stand-alone package separate from the one handling reference frames and other rotations. Although I'm afraid that the precision you mention might not be applicable to all timescale transformations. As a basic example, moving from UTC to UT1 (which is required for the ITRF-to-GCRF rotation) inevitably passes by the EOP DUT1 corrections, which have inherent uncertainties in the order of few microseconds. At the same time, for certain applications, when extremely high accuracy is required, the concept of space and time become bonded, such that the transformations between timescales must account for the actual observers locations, which in turn might depend upon the reference frames in which those positions are represented. As a side note: our needs are mainly related to astrodynamic and GNC applications, so we might definitely have more relaxed requirements on accuracy.

We currently represent time as a the sum of an integer number of seconds since J2000 plus the fraction of seconds (which can be expressed in any custom type, i.e. BigFloat), so technically I guess we would only require more precise algorithms between TAI to UTC (which currently uses ERFA routines) as well as improved routines to convert from Gregorian to Julian Days and viceversa (?).

The dependency graph we have in mind within JuliaSpaceMissionDesign is quite simple:
Tempo (Timescale Transformations) -> Packages that provide rotations between specific reference frames (e.g., ITRF-to-GCRF, IAU body-fixed frames, etc..) -> FrameTransformations (which handles the chain of rotations and translations between all the inserted frames).

Although I'm afraid I won't have the time to develop\investigate\test different algorithms to improve SOFA accuracy, I'd be open to discuss how the models we have already implemented can be further improved\adapted to your needs, so suggestions are more than welcome!

[barrettp]

1. I don't think memory allocation is big hit for 1000+ targets if it is minimized to a few of allocations, but I could be wrong.

2. The secondary goal is not high on my list, but it would be interesting to investigate when time allows. In this respect it should be noted that SOFA/ERFA functions are use constants with errors as per the original references. However, the SOFA/ERFA functions assume that the constants are exact. It would be interesting to use the Measurements package to propagate those errors through the calculation to see how accurate they actually are. I suspect they are not as accurate as the authors or users assume because of this issue.

3. I think the SOFA/ERFA header files sofam.c & erfam.c provide a good starting point for grouping the various functions. The rotation, angle and similar functions, what I call the utility functions, and possibly the combined timescale and reference-frame functions are at the bottom of the dependency tree. The various Earth models are next up the tree and the astrometry models above that.

4. My vision of the timescale dependency tree is value -> measurement -> unit -> epoch/offset -> timescale, where the epoch/offset refers the T0 reference date, i.e., JD, MJD, J2000.0, etc.

5. I'm not enamored with defining the units of the timescale to be only seconds, because there are many algorithms that use units of days, centuries, and occasionally millennia. I'd prefer that these conversions be done automatically by the algorithm. The current AstroTime module stores all dates as seconds, even though the associated units can be any time unit. Thus the unit does not necessarily agree with the value.

6. Concerning high precision timescale calculations, this would likely be an optional feature and could be done in several ways such as decimal arithmetic, BigFloats, or both.

7. It looks like the first module to work on is the timescale module, since that is very fundamental to all the others. So, I think we should agree on it first.

Let me know how you would like to proceed.

[andreapasquale94]

Hello @barrettp,

I appreciate your detailed insights into the ongoing discussion.
I'd like to contribute to the conversation and try to bring some clarity to the various aspects being discussed.

---

Firstly, I would strongly invite you to explore our organization, JuliaSpaceMissionDesign, and the associated packages. Our primary focus is on the development and testing of modern code for satellite trajectory design and navigation.
In alignment with our objectives, we have prioritized the following key aspects:

• Heap Allocation-Free Code: Given the nature of our applications, we are keen on minimizing CPU time spent on heap allocations, as it constitutes a significant performance consideration for us.
• Intuitive UI: We aim to create a user-friendly experience through clear function namings and logically grouped functionalities across different packages.
• Thread-Safety: It's crucial for us to ensure thread-safety, enabling the use of our packages in distributed systems.

You can also find more details about the organization in this article or visiting our documentation page.

---

While we acknowledge the value of developing a drop-in replacement for SOFA/ERFA functions, we would like to highlight that our vision for our Julia packages involves leveraging the full capabilities and advantages offered by the Julia language. This may potentially deviate from the goal of creating a direct replacement. I would appreciate your thoughts on this and whether this aligns with your intended target.

As @MicheleCeresoli suggested, we could potentially utilize the existing implementations in some packages of the JuliaSpaceMissionDesign ecosystem to support the functionalities of Astrometry.jl. Specifically, I believe we could effectively use Tempo.jl and IERS.jl for all time and Earth-associated reference frame transformations, and possibly FramesTransformations.jl. The following figure illustrates this:

In this figure, all the packages highlighted in green and yellow are part of the JSMD ecosystem, the grey ones are Julia community packages, and a potential dependency graph for Astrometry.jl (shown in purple) is depicted.

---

Please note that thanks to Julia multiple dispatch, many aspects you mentioned like:

• My vision of the timescale dependency tree is value -> measurement -> unit -> epoch/offset -> timescale ...
• I'm not enamored with defining the units of the timescale to be only seconds ... I'd prefer that these conversions be done automatically by the algorithm ...
• Concerning high precision timescale calculations, ... decimal arithmetic, BigFloats, or both.

could be implemented on top of some low-level functions, which, in turn, would be optimized for execution speed and deeply tested. This is the approach we are following in our organization and we found it very flexible, scalable, and composable.

For example, the units handling could be a package extension of Astrometry.jl that is loaded whenever the Unitful.jl package is loaded. The same could be done with Measurements.jl.

---

The secondary goal is not high on my list, but it would be interesting to investigate when time allows. In this respect it should be noted that SOFA/ERFA functions are use constants with errors as per the original references. However, the SOFA/ERFA functions assume that the constants are exact. It would be interesting to use the Measurements package to propagate those errors through the calculation to see how accurate they actually are.

At the moment this is something we cannot support with our packages, as it was not of interest in our applications. I investigated a bit the options we would have here and I believe this could be cool IERS.jl package exstension. We could definitely provide support in that in case you have some time to work on it.

Summary

We would definitely be interested in supporting you in interfacing Astrometry.jl with IERS.jl and Tempo.jl and in the development of their eventual Measurements.jl extensions.

I believe a package for astronomers would be a wonderful addition to what we already have within our ecosystem and we could both benefit from having composable packages.

[barrettp]

It's all about time

Literally and metaphorically, it's all about time. None of the other modules matter without a complete and comprehensive time module. Unfortunately, I do not find Tempo to be that module. It relies solely on seconds as the unit, which I believe to be a limitation. The two fundamental modules are Timescales and Reference-frames, since a timescale is one aspect of a reference frame, e.g., geocenteric and barycenteric. They are the roots of the tree. All other modules depend on these two. Therefore, I think that they should be generic and should be implemented to handle all use cases of time, including the high precision of atomic, optical, and nuclear clocks. The intention is to make it the ultimate time module for physics. Otherwise, there will be a proliferation of time modules designed for particular niche applications. I see no need for that. Tempo's micro- or picosecond precision may be fine for your IERS application, but it does not satisfy the femto- or zeptosecond precision of master clocks and pulsar astronomy.

It appears that your modules are too focused on geocentric IERS applications. I would like to take a wider view that places all reference frames on equal footing. The primary reference frame for astronomers is barycentric. Geocentric and geographic reference frames are just nuisance reference frames to us. It depends on your perspective. So, I get the impression that your dependency graph does not see the big picture. Note that I worked at the US Naval Observatory in the Earth Orientation, Astrometry, and Time Services departments. So, although I am not an expert on all of these disciplines, I have a good understanding of how they fit together, and I know that time is the key component. Also, IERS's applicability is severely limited without astrometry providing DeltaUT1. No reference frame or timescale is dominant, except may TAI. This is the reason that I keep harping on a comprehensive time module.

If you are prepared to modify Tempo so that it satisfies my requirements, then I think we have a way forward. Otherwise, not so much. I understand you not wanting to do so, since you have spent much time in optimizing the performance of your modules. Although performance is important to me, it takes a back seat to precision.

Requirements

To reiterate, my requirements are: 1) unit agnostic, 2) measurement compatible, 3) timescale and/or reference frame agnostic, and 4) optional high precision. (I think that covers it.)

[andreapasquale94]

Thank you again for clarifying, now is much clearer what your objectives and requirements are.

I would start with some brief comments:

The two fundamental modules are Timescales and Reference-frames, since a timescale is one aspect of a reference frame, e.g., geocenteric and barycenteric. They are the roots of the tree. All other modules depend on these two.

This is entirely in line with the approach we have in Tempo and FrameTransformations, with the first one implementing frame-agnostic routines and the second time-agnostic ones. These two should be "combined" at user level, depending on the application (for example, in our case we usually use GCRF and TT as roots).

Tempo's micro- or picosecond precision may be fine for your IERS application, but it does not satisfy the femto- or zeptosecond precision of master clocks and pulsar astronomy.

This is also understood. As you were saying, the need here would be to have a unit-agnostic epoch/time representation, so that time could be represented at arbitrary precision in different time units. Is this, summarizing, what you meant?

It appears that your modules are too focused on geocentric IERS applications. ... So, I get the impression that your dependency graph does not see the big picture.

Yes, this is correct if you look at the IERS.jl package.
For our applications, the GCRF and the geocentric associated frames are usually enough.
However, within the FrameTransformations.jl, we do implement already some barycentric frames (ICRF, ecliptic, galactic, fk4, fk5...) as well as the basic methods and interfaces to easily build any additional desired frame.

To conclude, let me take some time to evaluate if/how we can accomodate the requirements you mention and see what could be the way forward. I definitely would like to find a way to try to integrate your needs with our infrastructure, if it is non-breaking for the higher level packages.

[barrettp]

Yes, a unit agnostic epoch/time representation with the potential or option for arbitrary precision.

I'd just like to reiterate that the SOFA module is not meant to be the primary astrometry module. It is secondary and is intended to be uses to check the correctness of the primary module. The primary module currently contains implementations of nutation, precision, and such algorithms with a minimal set of functions. I currently use it to apply barycentric corrections to time-series observations of astronomical objects.

[andreapasquale94]

I'd just like to reiterate that the SOFA module is not meant to be the primary astrometry module.

Apologies but this is now not clear again. My understanding is that the intension here is to provide the users a set of astrometry-related functions compatible with the current standard's (erfa/sofa). The way in which these are organised doesn’t really matters for us. This is of interest for us, since we don’t have those functionalities in our ecosystem.

The primary module currently contains implementations of nutation, precession…

This is instead what we were proposing to leverage from our ecosystem, as we do already implement, optimise, and deeply test those functionalities in our packages.

Can you please clarify?
Also, @MicheleCeresoli do you have a different understanding?

[barrettp]

The Astrometry module will contain a necessary and sufficient set of astrometry functions when it is complete. However, it also contains the SOFA submodule that is a complete Julia implementation of the SOFA/ERFA functions. It can be used as a drop-in replace for the wrapped C version and to ensure that the main Astrometry functions are implemented properly.

So, in summary, there is Astrometry.precess, Astrometry.nutate, etc. and then there is Astrometry.SOFA.nut00, etc.

[MicheleCeresoli]

I think a lot of things have been laid on the table but I am afraid there still might be some confusion. I'll start from the last things that have been said and move on to touch all the remaining topics.

SOFA/ERFA and IERS-related functions

However, it also contains the SOFA submodule that is a complete Julia implementation of the SOFA/ERFA functions. It can be used as a drop-in replace for the wrapped C version and to ensure that the main Astrometry functions are implemented properly.

With respect to the SOFA functions, from this conversation I identify all the following needs/requirements (some coming from your comments, some from ours):

1. Have a pure-Julia implementation of the SOFA/ERFA functions.
2. Optimise the execution times when iterating over a large set of objects.
3. Support high-precision operations (e.g., Float128, BigFloats).
4. Support error propagation to assess the accuracy of the transformations.
5. Support automatic differentiation algorithms.

When I first saw this package, my initial thoughts were to propose you the possibility to leverage the package that we developed rather than having another Julia implementation of the SOFA functions. At the moment IERS completely satisfies objectives (1), (3) and (5), while also providing some additional functions related to the IERS conventions that are not present in the ERFA/SOFA libraries. Requirement (4) is not satisfied: the type of the output is perfectly compatible with that of the input, but we also assume all the constants are exact (although I believe the same assumption has currently been made in the version contained in the Astronometry.SOFA module). Finally, regarding requirement (2), as I stated in a previous post, we optimised execution-times by minimising the number of repeated\useless calculations but do not support vectorised code for a variety of reasons:

• Avoid memory allocations: our typical applications do not require simultaneously iterating over million of objects. Thus, when you consider a few iterations, memory allocation has quite the impact because the execution times are already low.

• Julia for loops are already quite efficient and with the benchmarks we performed on our optimised functions (IAU2006/2000A), a for loop over 1 million objects would take approximately 40 seconds on a basic Intel Core i7-6700 @ 3.4 GHz.

I understand the SOFA sub-module is not meant to be the primary module, although I struggle to see the difference between the two, since at the moment the precession, nutation etc.. functions appear to be quite equal to their SOFA counterparts. With this in mind, my proposal is again to check whether IERS can be partially used for this: it exposes a standard set of interfaces that are implemented for each IERS convention (1996, 2003 and 2010) and could definitely provide at the very least the replacement for the C-version (i.e., substituing what you call the SOFA submodule). It also exposes a set of routines to compute the rotations between various ECI\ECEF frames but I'm also aware that you've already written all of the SOFA functions, and additionally if your goal is to change those functions, (e.g., by adding the measurement uncertainties) it would not be compatible with your primary module needs.

Overall I would be up to slightly modify the package to incorporate additional needs (e.g., computation of specific quantities that are currently missing) but I do not plan on adding support for vectorised code and\or measurement uncertainties so I completely understand if you were to stick to your current implementation.

So, although I am not an expert on all of these disciplines, I have a good understanding of how they fit together, and I know that time is the key component. Also, IERS's applicability is severely limited without astrometry providing DeltaUT1.

Small note on this (I'll get back to time-handling at the end of the post). IERS is meant to extend the timescale graph implemented within Tempo by providing the UTC to UT1 transformation using the data contained in the EOP files, as commonly done in our field.

Reference Frames

It appears that your modules are too focused on geocentric IERS applications. I would like to take a wider view that places all reference frames on equal footing. The primary reference frame for astronomers is barycentric.

Actually, one of the key points of FrameTransformations.jl is the capability to support (almost) any type of reference frame. So that if you primarily work with the BCRS, you can simply define that to be the root frame of your reference frame graph (and associate to it whatever timescale you need). IERS is only a small part of the entire ecosystem. It is a package that we've recently created as part of a general rework of FrameTransformations, that is only meant to provide the user the possibility to add Earth-related frames (plus compute a ton of other IERS-related quantities, if needed). In a similar fashion, within FrameTransformations.Orient we also have routines that allow you to create/add IAU-body fixed frames, but that does not mean that the we only focus on those frames. Rather, our final goal is to be capable of supporting any type of frame (which is the reason why I am very interested on your possible needs\applications and see what are the limitations on our side).

Time Handling

I think there's two concepts at play here:

1. Absolute Time: a generic epoch that represents a specific instant in time in any desired timescale.
2. Relative Time: the time between two generic epochs, e.g., time passed since J2000.

For Absolute Time a distinction must be made between the way time is internally represented and the way it is returned to the user. Within Tempo we made the decision to internally store all absolute times as an integer number of seconds elapsed since J2000 plus a a fraction of seconds. The latter argument can be provided in any type so that theoretically we already support arbitrary precision numbers. I'm really not sure into what your requirement on supporting high precision atomic, optical or nuclear clocks technically translates. If you are talking about the capability to distinguish between two epochs with differences in the order of zeptoseconds, I think we can already cover that by for example expressing the fraction of seconds as a BigFloat number.

On the other hand, if you are talking about the capability to convert a single epoch between different timescales with the same zeptosecond precision, I don't really see how that can be achieved for all timescales, since the transformations between most of them have uncertainties that are way higher than that (e.g., the UTC-to-UT1 example).

This is all the internal stuff.

The epoch constructors we adopted already allow the user to specify an absolute time with respect to the most common epoch offsets (e.g., JD, J2000, MJD) in any generic timescale. Internally, that will simply be converted to seconds since J2000. Then at this point, the user is free to convert his generic epoch to a relative time in the unit he desires. For example, at the moment we export functions that automatically transform an epoch into the number of seconds, days, years or centuries since J2000 and we could easily add other public interfaces for the most common epoch offsets (e.g., JD and MJD).

The big INTENTIONAL difference with respect to AstroTime is that we do not use singletons or types to express relative times. Meaning that when you convert an epoch to seconds since J2000, your output type will simply be a scalar number (e.g., Float64). Although this requires a little more attention on the user side, it allows to write higher performance algorithms.

Thus, to iterate over your requirements:

1. Unit agnostic: the internal representation is fixed but the user is free to retrieve the time elapsed between his epoch and a fixed subset of reference epochs in a subset of units (e.g., milliseconds, seconds, days, centuries...). However, the output type is not expressed with a dedicated unit type.
2. Timescale agnostic: the epoch can be represented in any desired timescale.
3. High precision: absolute time can be expressed in any desired precision. However, that precision might be lost when converting to other timescales due to the accuracy of the timescale transformation.

I will once again say that I'm open to slight modifications of Tempo to increase its flexibility (e.g., how other epoch\offsets can be defined or on the epoch constructors etc...).

I don't know whether this clarifies some of the doubts that have arisen. I undestand that our needs\vision might be slightly different but let me know if you think that there's still a way forward.

Thanks for taking the time to read through this!

[barrettp]

Thanks for the detailed explanation. I think my next step is to see if Tempo satisfies my needs and long term vision. I certainly don't want to reinvent the wheel if it's not necessary. It may take a few weeks to investigate this issue, so don't expect a response anytime soon.

I should also investigate the vectorization issue. My understanding is that for large datasets, vectorized code has higher performance than non-vectorized code. The use case that I am thinking of is the Gaia star catalog that has >10e9 stars. It is conceivable that a user may want to process (i.e., precess) a large fraction (>10e6) of those stars to a new epoch. I am interested to see the relative difference in performance between the two implementations.

The desire for high precision is for differences in epochs or times using a single timescale, not between timescales. I know that there is typically a loss of precision between timescales. I'm primarily thinking TAI, but this may also apply to pulsar timing studies where the timing precision is less than a microsecond and the datasets span more than 50 years. That requires a precision of <1e-16.

To reiterate, the Julia version of the SOFA library is essentially a compatibility library for those that are currently using the C version of the library. It enables them to migrate easily to a pure Julia version without changing their code with the eventual goal of using a simpler and modern Julia Astrometry module. The C version of the SOFA library is based on a nearly direct implementation of the FORTRAN version that uses an archaic (40 year old) programming style. As a result, it is not easy for novices to use and, therefore, its use should be discouraged. I wouldn't worry about it too much.

[tamasgal]

Hey there! Are there any updates on this? I'd like to integrate this into our software pipelines but I am not sure what the status is 😉

[barrettp]

There is a problem with the nightly CompatHelper script that causes it to fail. The result is that the package cannot be registered. I have no idea where this script came from and why it is doing what it is doing. I've spent a few hours investigating the issue, but have been to busy with other issues to fix it. Once I get a few papers submitted and version 0.2 of the FITS package implemented, I may have some time next week to work on it some more. If you can fix the CompatHelper issue, then that may solve your problem.

[tamasgal]

I cannot help there due to lack of access, but I am pretty sure that it can be solved via activating "Allow GitHub Actions to create and approve pull requests" here:

https://github.com/barrettp/Astrometry.jl/settings/actions

That's at least what I read here in the CI logs: https://github.com/barrettp/Astrometry.jl/actions/runs/13447866627/job/37576927138#step:6:105

Server: github.com

{"message":"GitHub Actions is not permitted to create or approve pull requests.","documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"403"}""", HTTP.Exceptions.StatusError(403, "POST", "https://api.github.com/repos/barrettp/Astrometry.jl/pulls", HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden

[barrettp]

Thank you. I made the change. Now to see if it solves the problem.

[tamasgal]

It worked: #4

Let me know if I can be of any help 🙂

[barrettp]

Fixed the SOFA/bp06 test failure by splitting one test into three and also fixed several CI workflow problems.