Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres ranges rebuild #1017

Merged
merged 32 commits into from
May 10, 2024
Merged

Postgres ranges rebuild #1017

merged 32 commits into from
May 10, 2024

Conversation

SpacemanPaul
Copy link
Contributor

@SpacemanPaul SpacemanPaul commented May 8, 2024

Complete rewrite of the OWS schema management:

  1. All tables and views moved to the new "ows" schema.
  2. Cleanup tool for removing the old "wms" schema (and the old MV's which lived in the public schema)
  3. Product and multiproduct range tables combined into a single "layer ranges" table. Unused historic sub-product table removed.
  4. Stronger permissions management. Can now separately grant read-only (service running user) permissions and read-write permissions (datacube-ows-update runnning users).
  5. Note that manipulating the schema now generally requires dbo/superuser permissions.
  6. Range table updating and reading code completely rewritten and greatly simplified.
  7. Misc other cleanup (including renaming confusing "product" references to "layer".

Still in draft to get the integration tests running on github again and check test coverage - some documentation updates also required.


📚 Documentation preview 📚: https://datacube-ows--1017.org.readthedocs.build/en/1017/

Copy link

codecov bot commented May 9, 2024

Codecov Report

Attention: Patch coverage is 89.08451% with 31 lines in your changes missing coverage. Please review.

Project coverage is 93.59%. Comparing base (722bc6e) to head (0de93b4).
Report is 34 commits behind head on develop-1.9.

Files with missing lines Patch % Lines
datacube_ows/update_ranges_impl.py 80.64% 18 Missing ⚠️
datacube_ows/ows_configuration.py 84.90% 8 Missing ⚠️
datacube_ows/product_ranges.py 95.40% 4 Missing ⚠️
datacube_ows/wms_utils.py 95.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff               @@
##           develop-1.9    #1017      +/-   ##
===============================================
- Coverage        93.77%   93.59%   -0.19%     
===============================================
  Files               46       46              
  Lines             6573     6616      +43     
===============================================
+ Hits              6164     6192      +28     
- Misses             409      424      +15     
Files with missing lines Coverage Δ
datacube_ows/cfg_parser_impl.py 98.68% <100.00%> (ø)
datacube_ows/loading.py 80.81% <100.00%> (ø)
datacube_ows/mv_index.py 94.05% <ø> (ø)
datacube_ows/ogc.py 86.02% <100.00%> (+1.06%) ⬆️
datacube_ows/styles/base.py 95.91% <100.00%> (ø)
datacube_ows/wcs1.py 100.00% <100.00%> (ø)
datacube_ows/wcs1_utils.py 95.25% <100.00%> (ø)
datacube_ows/wcs2.py 92.30% <100.00%> (ø)
datacube_ows/wcs2_utils.py 91.95% <100.00%> (ø)
datacube_ows/wms_utils.py 92.05% <95.00%> (-0.03%) ⬇️
... and 3 more

@SpacemanPaul SpacemanPaul marked this pull request as ready for review May 9, 2024 04:15
@SpacemanPaul SpacemanPaul requested review from Ariana-B and omad and removed request for omad and Ariana-B May 9, 2024 04:16
@SpacemanPaul SpacemanPaul marked this pull request as draft May 9, 2024 04:36
@SpacemanPaul SpacemanPaul requested review from omad and Ariana-B May 9, 2024 05:57
CONTRIBUTING.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
check-code-all.sh Outdated Show resolved Hide resolved
datacube_ows/loading.py Outdated Show resolved Hide resolved
datacube_ows/wms_utils.py Outdated Show resolved Hide resolved
datacube_ows/wms_utils.py Outdated Show resolved Hide resolved
docs/database.rst Outdated Show resolved Hide resolved
tests/test_no_db_routes.py Outdated Show resolved Hide resolved
@SpacemanPaul
Copy link
Contributor Author

Thanks Peter. That was a really helpful review.

Copy link

@Ariana-B Ariana-B left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

datacube_ows/update_ranges_impl.py Show resolved Hide resolved
Copy link
Contributor

@pjonsson pjonsson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're in opposite time zones so me blocking progress is probably worse than any errors slipping in, go ahead and merge according to your own judgement.

I don't know anything about the code base, but besides the assignment to TYPE_CHECKING, the changes look reasonable to me.

@SpacemanPaul SpacemanPaul merged commit d1ac065 into develop-1.9 May 10, 2024
10 checks passed
@SpacemanPaul SpacemanPaul deleted the postgres-ranges-rebuild branch May 10, 2024 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants