[epic] Springclean and update Giftless (2023/2024) - for latest python etc

[rufuspollock]

Giftless has not had a makeover since ~2020/2021 so an update is in order and @athornton is leading the charge 🙏

Acceptance

Updates

Beta Give feedback

1. Have it work with modern python versions ✅2023-12-28 in Work correctly with pyjwt 2.x #124 and Modernize giftless #125
2. Lint with a modern linter e.g. black Replace isort/flake8/black with ruff: tickets/DM-42231B #130
3. Maintainable test suite mock Google Cloud Storage: tickets/DM-42231D #136
4. Build/push docker artifacts via GHA, with matching tags in GH and Docker Hub #174
   +0
5. Or, just (or additionally) push to ghcr.io, which simplifies authentication somewhat Github Actions revamp: tickets/DM-42231C #135
6. generally spiffing up pre-commit hooks and GHA Github Actions revamp: tickets/DM-42231C #135
7. Update docs to flag support for AWS etc

Tasks

Tasks

Beta Give feedback

1. Work correctly with pyjwt 2.x #124
   +0
2. Modernize giftless #125
   +0

[athornton]

Moving in the right direction: a lot of linting cleanup is done, and there's a PR to use Ruff in flight. I picked Ruff because it's what Rubin Observatory is now using internally, and it replaces all of flake8, black, and isort all at once, which is very handy. It's extremely opinionated, but also very configurable. I'm using basically the same set of rules we use, which might not be quite what Datopian wants, but it's a place to start.

After that I'm going to modernize Github actions and try to get the Google storage test working again, which will mean dropping pytest-vcr from it. After that if I have remaining time budget I might try to do the same for the AWS and Azure storage tests, but at least those are still working for now.

[athornton]

Github Actions are updated too with a PR to add those also in flight.

Mostly, if I'm being honest, because I'm avoiding diving into the test suite, which I will tackle next week.

[athornton]

#134 was exchanged for #135.

#136 is now open, and that at least touches everything but the doc updates. The current state (with #136 merged) isn't great, in that the mock basically just turns Google storage into local storage, but I am worried that I couldn't get a client that mocked HTTP responses working by the end of the week, which is all the time I have.

[athornton]

I'm going to need Datopian help for:

1. making the ghcr.io package repository public
2. acquiring Docker login credentials to push to Docker Hub
3. setting up PyPi trusted publishers.

Once we've done that, we can cut a new release. I think this one should be 0.6.0 because it's a pretty big change from the previous version.

[rufuspollock]

@anuveyatsu or @demenech could you take a quick look at @athornton requests above - that will allow us to automate things going forward.

[athornton]

Just bumping this. It won't take very long, although actually scheduling a 30 minute Zoom-or-equivalent call for me to talk whoever on the Datopian side through the various tasks may be more efficient than GitHub comments or conversation on the Discord. I have a little availability this week and then I'm at a conference next week.

[rufuspollock]

@demenech is there a chance you can schedule this with @athornton (via discord is probably easiest). Then we can have this setup 😄

@athornton thanks for all the forward progress! 👏

[demenech]

hi @rufuspollock

The issue on my end is that I don't have the necessary credentials i.e. Dockerhub and PyPi credentials.

If these are provided to me I'm glad to help.

[rufuspollock]

@demenech I would ask @anuveyatsu then. We may no longer publish to dockerhub so other options are possible. Pypi is crucial one and we need to look into the datopian account credentials there.

[devinnasar]

Does this refresh cover the signing version that I mention in #106?

[rufuspollock]

@devinnasar it could if you get a PR in 🙂

[rufuspollock]

@athornton i've split out pypi release to #173 and docker stuff to #174. I suggest we close this as fixed (i'll leave you with the honors) and prioritize those two.