Approach to CI for Firedrake-dependent packages

[tommbendall]

Posting here at David's suggestion...

I've been thinking about how to best manage Gusto’s CI, but this is probably also relevant to other packages like Thetis, asQ and icepack. At the moment we run Gusto CI against firedrake-vanilla:latest, and test our main branch every Monday morning.

In the last couple of months, there have been quite a few different changes to Firedrake and its underlying packages that have broken our tests, and we’ve been struggling to keep up!

To try to make things easier, we've been wondering it would make more sense for us to be testing against older stable versions of Firedrake?

Or to help us to allocate time to adopt changes, I was wondering if developers might be willing to announce (e.g. on Slack) any Firedrake API or interface changes that might break downstream code. We would also definitely welcome people testing their firedrake/ufl/etc branches in Gusto, if they think this might help show up errors that wouldn’t be caught by Firedrake’s test suite!

[dham]

We do preannounce intentional API changes, and we often carry around warnings for a very extended period before we change features (e.g. the interpolation change which has been warning for well over a year now).

I think there was only one unannounced API change recently, which was an accident caused by a developer bypassing code review in a way that shouldn't have happened. We try to avoid that happening.

Some other things that have suddenly broken in recent times have been due to upstream dependencies changing (e.g. loopy and its dependencies, or PETSc). We don't have a lot of control over that happening.

Currently there is no concept of a more stable Firedrake. If you pin to an older version, there is a high risk of that randomly stopping working due to a dependency going out of sync (this is the problem that people most often have if they try to run from a Zenodo archive). We do plan to start versioned releases of Firedrake later this year once:

• The change to pip installation is through.
• The PETSc issue that causes us to diverge from PETSc main is fixed.
  When that happens we will tie each Firedrake release to a particular PETSc release, which reduces the risk of PETSc going out of sync. We are still working out how to deal with loopy versioning.

[tommbendall]

Thanks for the reply David, using versioned releases of Firedrake sounds like a potential option in the future then. The main thing from our side is just trying to anticipate when we'll need to make fixes in response to upstream changes