Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test db build #1021

Merged
merged 3 commits into from
May 21, 2024
Merged

Test db build #1021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a5f6a9
Attempt to index S2 L2A us-west data in GH test harness.
SpacemanPaul May 20, 2024
ed69d3f
Iteratively get working on github.
SpacemanPaul May 20, 2024
71c9528
Delete unused database dumps.
SpacemanPaul May 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 79 additions & 4 deletions check-code-all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,18 @@ set -ex

# ensure db is ready
sh ./docker/ows/wait-for-db
# Run tests, taking coverage.
# Users can specify extra folders as arguments.

# Initialise ODC schema

datacube system init

# Add extended metadata types

datacube metadata add https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/product_metadata/eo3_landsat_ard.odc-type.yaml
datacube metadata add https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/product_metadata/eo3_sentinel_ard.odc-type.yaml

# On top of base sql sample, index two more product for multiproducts
# Test products
datacube product add ./integration_tests/metadata/s2_l2a_prod.yaml
datacube product add https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/products/baseline_satellite_data/c3/ga_s2am_ard_3.odc-product.yaml
datacube product add https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/products/baseline_satellite_data/c3/ga_s2bm_ard_3.odc-product.yaml
datacube product add https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/products/land_and_vegetation/c3_fc/ga_ls_fc_3.odc-product.yaml
Expand All @@ -23,6 +28,75 @@ datacube product add https://raw.githubusercontent.com/GeoscienceAustralia/dea-c

datacube product add https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/products/baseline_satellite_data/geomedian-au/ga_ls8c_nbart_gm_cyear_3.odc-product.yaml

# S2 datasets from us-west-2 (might not work)
MDL=./integration_tests/metadata
python ${MDL}/metadata_importer.py <<EOF
${MDL}/s2_l2a_ds_01.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XD/2021/12/S2B_51LXD_20211231_0_L2A/S2B_51LXD_20211231_0_L2A.json
${MDL}/s2_l2a_ds_02.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YE/2021/12/S2B_51LYE_20211231_0_L2A/S2B_51LYE_20211231_0_L2A.json
${MDL}/s2_l2a_ds_03.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2021/12/S2B_51LXF_20211231_0_L2A/S2B_51LXF_20211231_0_L2A.json
${MDL}/s2_l2a_ds_04.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2021/12/S2B_51LXE_20211231_0_L2A/S2B_51LXE_20211231_0_L2A.json
${MDL}/s2_l2a_ds_05.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WD/2021/12/S2B_51LWD_20211231_0_L2A/S2B_51LWD_20211231_0_L2A.json
${MDL}/s2_l2a_ds_06.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YF/2021/12/S2B_51LYF_20211231_0_L2A/S2B_51LYF_20211231_0_L2A.json
${MDL}/s2_l2a_ds_07.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2021/12/S2B_51LWE_20211231_0_L2A/S2B_51LWE_20211231_0_L2A.json
${MDL}/s2_l2a_ds_08.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2021/12/S2B_51LWF_20211231_0_L2A/S2B_51LWF_20211231_0_L2A.json
${MDL}/s2_l2a_ds_09.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YD/2021/12/S2B_51LYD_20211231_0_L2A/S2B_51LYD_20211231_0_L2A.json
${MDL}/s2_l2a_ds_10.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2022/1/S2B_51LWF_20220110_0_L2A/S2B_51LWF_20220110_0_L2A.json
${MDL}/s2_l2a_ds_11.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YD/2022/1/S2B_51LYD_20220110_0_L2A/S2B_51LYD_20220110_0_L2A.json
${MDL}/s2_l2a_ds_12.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XD/2022/1/S2B_51LXD_20220110_0_L2A/S2B_51LXD_20220110_0_L2A.json
${MDL}/s2_l2a_ds_13.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WD/2022/1/S2B_51LWD_20220110_0_L2A/S2B_51LWD_20220110_0_L2A.json
${MDL}/s2_l2a_ds_14.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2022/1/S2A_51LXE_20220108_0_L2A/S2A_51LXE_20220108_0_L2A.json
${MDL}/s2_l2a_ds_15.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WD/2022/1/S2A_51LWD_20220105_0_L2A/S2A_51LWD_20220105_0_L2A.json
${MDL}/s2_l2a_ds_16.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2022/1/S2B_51LXE_20220110_0_L2A/S2B_51LXE_20220110_0_L2A.json
${MDL}/s2_l2a_ds_17.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2022/1/S2B_51LWE_20220110_0_L2A/S2B_51LWE_20220110_0_L2A.json
${MDL}/s2_l2a_ds_18.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YE/2022/1/S2B_51LYE_20220110_0_L2A/S2B_51LYE_20220110_0_L2A.json
${MDL}/s2_l2a_ds_19.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2022/1/S2B_51LXF_20220110_0_L2A/S2B_51LXF_20220110_0_L2A.json
${MDL}/s2_l2a_ds_20.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2022/1/S2B_51LXF_20220103_0_L2A/S2B_51LXF_20220103_0_L2A.json
${MDL}/s2_l2a_ds_21.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YF/2022/1/S2B_51LYF_20220110_0_L2A/S2B_51LYF_20220110_0_L2A.json
${MDL}/s2_l2a_ds_22.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2022/1/S2A_51LWE_20220105_0_L2A/S2A_51LWE_20220105_0_L2A.json
${MDL}/s2_l2a_ds_23.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2022/1/S2A_51LWE_20220108_0_L2A/S2A_51LWE_20220108_0_L2A.json
${MDL}/s2_l2a_ds_24.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XD/2022/1/S2A_51LXD_20220105_0_L2A/S2A_51LXD_20220105_0_L2A.json
${MDL}/s2_l2a_ds_25.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2022/1/S2A_51LXF_20220108_0_L2A/S2A_51LXF_20220108_0_L2A.json
${MDL}/s2_l2a_ds_26.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YE/2022/1/S2A_51LYE_20220105_0_L2A/S2A_51LYE_20220105_0_L2A.json
${MDL}/s2_l2a_ds_27.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2022/1/S2B_51LXE_20220103_0_L2A/S2B_51LXE_20220103_0_L2A.json
${MDL}/s2_l2a_ds_28.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2021/12/S2A_51LXE_20211226_0_L2A/S2A_51LXE_20211226_0_L2A.json
${MDL}/s2_l2a_ds_29.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YD/2022/1/S2A_51LYD_20220105_0_L2A/S2A_51LYD_20220105_0_L2A.json
${MDL}/s2_l2a_ds_30.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2021/12/S2A_51LXE_20211229_0_L2A/S2A_51LXE_20211229_0_L2A.json
${MDL}/s2_l2a_ds_31.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XD/2021/12/S2A_51LXD_20211226_0_L2A/S2A_51LXD_20211226_0_L2A.json
${MDL}/s2_l2a_ds_32.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2021/12/S2B_51LXE_20211224_0_L2A/S2B_51LXE_20211224_0_L2A.json
${MDL}/s2_l2a_ds_33.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2022/1/S2A_51LWF_20220108_0_L2A/S2A_51LWF_20220108_0_L2A.json
${MDL}/s2_l2a_ds_34.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YF/2021/12/S2A_51LYF_20211226_0_L2A/S2A_51LYF_20211226_0_L2A.json
${MDL}/s2_l2a_ds_35.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YD/2021/12/S2B_51LYD_20211221_1_L2A/S2B_51LYD_20211221_1_L2A.json
${MDL}/s2_l2a_ds_36.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2021/12/S2B_51LXF_20211221_0_L2A/S2B_51LXF_20211221_0_L2A.json
${MDL}/s2_l2a_ds_37.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YF/2022/1/S2A_51LYF_20220105_0_L2A/S2A_51LYF_20220105_0_L2A.json
${MDL}/s2_l2a_ds_38.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2022/1/S2A_51LWF_20220105_0_L2A/S2A_51LWF_20220105_0_L2A.json
${MDL}/s2_l2a_ds_39.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2022/1/S2B_51LWE_20220103_0_L2A/S2B_51LWE_20220103_0_L2A.json
${MDL}/s2_l2a_ds_40.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2021/12/S2A_51LWF_20211229_0_L2A/S2A_51LWF_20211229_0_L2A.json
${MDL}/s2_l2a_ds_41.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2022/1/S2A_51LXE_20220105_0_L2A/S2A_51LXE_20220105_0_L2A.json
${MDL}/s2_l2a_ds_42.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2021/12/S2A_51LWF_20211226_0_L2A/S2A_51LWF_20211226_0_L2A.json
${MDL}/s2_l2a_ds_43.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2021/12/S2B_51LXF_20211224_0_L2A/S2B_51LXF_20211224_0_L2A.json
${MDL}/s2_l2a_ds_44.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2022/1/S2A_51LXF_20220105_0_L2A/S2A_51LXF_20220105_0_L2A.json
${MDL}/s2_l2a_ds_45.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2022/1/S2B_51LWF_20220103_0_L2A/S2B_51LWF_20220103_0_L2A.json
${MDL}/s2_l2a_ds_46.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YE/2021/12/S2A_51LYE_20211226_0_L2A/S2A_51LYE_20211226_0_L2A.json
${MDL}/s2_l2a_ds_47.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2021/12/S2B_51LWF_20211221_0_L2A/S2B_51LWF_20211221_0_L2A.json
${MDL}/s2_l2a_ds_48.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YF/2021/12/S2B_51LYF_20211221_0_L2A/S2B_51LYF_20211221_0_L2A.json
${MDL}/s2_l2a_ds_49.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2021/12/S2A_51LXF_20211229_0_L2A/S2A_51LXF_20211229_0_L2A.json
${MDL}/s2_l2a_ds_50.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2021/12/S2A_51LWE_20211229_0_L2A/S2A_51LWE_20211229_0_L2A.json
${MDL}/s2_l2a_ds_51.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2021/12/S2A_51LWE_20211226_0_L2A/S2A_51LWE_20211226_0_L2A.json
${MDL}/s2_l2a_ds_52.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YD/2021/12/S2A_51LYD_20211226_0_L2A/S2A_51LYD_20211226_0_L2A.json
${MDL}/s2_l2a_ds_53.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XD/2021/12/S2B_51LXD_20211221_0_L2A/S2B_51LXD_20211221_0_L2A.json
${MDL}/s2_l2a_ds_54.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WD/2021/12/S2A_51LWD_20211226_0_L2A/S2A_51LWD_20211226_0_L2A.json
${MDL}/s2_l2a_ds_55.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WD/2021/12/S2B_51LWD_20211221_0_L2A/S2B_51LWD_20211221_0_L2A.json
${MDL}/s2_l2a_ds_56.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XF/2021/12/S2A_51LXF_20211226_0_L2A/S2A_51LXF_20211226_0_L2A.json
${MDL}/s2_l2a_ds_57.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2021/12/S2B_51LWE_20211224_0_L2A/S2B_51LWE_20211224_0_L2A.json
${MDL}/s2_l2a_ds_58.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WF/2021/12/S2B_51LWF_20211224_0_L2A/S2B_51LWF_20211224_0_L2A.json
${MDL}/s2_l2a_ds_59.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YE/2021/12/S2B_51LYE_20211221_0_L2A/S2B_51LYE_20211221_0_L2A.json
${MDL}/s2_l2a_ds_60.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2021/12/S2B_51LWE_20211221_1_L2A/S2B_51LWE_20211221_1_L2A.json
${MDL}/s2_l2a_ds_61.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/WE/2021/12/S2B_51LWE_20211221_0_L2A/S2B_51LWE_20211221_0_L2A.json
${MDL}/s2_l2a_ds_62.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2021/12/S2B_51LXE_20211221_1_L2A/S2B_51LXE_20211221_1_L2A.json
${MDL}/s2_l2a_ds_63.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/XE/2021/12/S2B_51LXE_20211221_0_L2A/S2B_51LXE_20211221_0_L2A.json
${MDL}/s2_l2a_ds_64.yaml https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/51/L/YE/2021/12/S2B_51LYE_20211221_1_L2A/S2B_51LYE_20211221_1_L2A.json
EOF

# S2 multiproduct datasets
datacube dataset add https://dea-public-data.s3.ap-southeast-2.amazonaws.com/baseline/ga_s2bm_ard_3/52/LGM/2017/07/19/20170719T030622/ga_s2bm_ard_3-2-1_52LGM_2017-07-19_final.odc-metadata.yaml --ignore-lineage
datacube dataset add https://dea-public-data.s3.ap-southeast-2.amazonaws.com/baseline/ga_s2bm_ard_3/52/LGM/2017/07/29/20170729T081630/ga_s2bm_ard_3-2-1_52LGM_2017-07-29_final.odc-metadata.yaml --ignore-lineage
Expand All @@ -47,7 +121,8 @@ datacube dataset add https://dea-public-data.s3.ap-southeast-2.amazonaws.com/der
datacube-ows-update --schema --write-role $DB_USERNAME
datacube-ows-update

# run test
# Run tests, taking coverage.
# Users can specify extra folders as arguments.
python3 -m pytest --cov=datacube_ows --cov-report=xml integration_tests/
cp /tmp/coverage.xml /mnt/artifacts

Expand Down
2 changes: 2 additions & 0 deletions datacube_ows/feature_info.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ def get_s3_browser_uris(datasets: dict[ProductBandQuery, xarray.DataArray],
if ds.crs != last_crs:
pt_native = pt.to_crs(ds.crs)
last_crs = ds.crs
else:
pt_native = pt
if ds.extent.contains(pt_native):
uris.append(ds.uris)
else:
Expand Down
2 changes: 1 addition & 1 deletion datacube_ows/ogc_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ def create_geobox(
if width is not None:
scale_x = (float(maxx) - float(minx)) / width
else:
scale_x = -scale_y
scale_x = -scale_y # pylint: disable=possibly-used-before-assignment
width = int(round((float(maxx) - float(minx)) / scale_x))
if height is None:
scale_y = - scale_x
Expand Down
1 change: 0 additions & 1 deletion docker/database/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
FROM kartoza/postgis:16
COPY s2_dump.sql /docker-entrypoint-initdb.d/
Loading
Loading