Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: standardise DEMs and allow retiling TDE-784 #142

Merged
merged 14 commits into from
Jul 30, 2023
Merged

feat: standardise DEMs and allow retiling TDE-784 #142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
22fcdcd
wip: add grouping (using templates) to standardising workflow
paulfouquet Jul 11, 2023
4fbfa2f
wip: add semaphore (commented out)
paulfouquet Jul 11, 2023
1bba94d
WIP: don't run basemaps when dem_lerc preset
MDavidson17 Jul 12, 2023
0286950
fix: add back semaphore and group remove scale from standardising
MDavidson17 Jul 19, 2023
937ec44
chore: add producers and licensors for elevation
amfage Jul 19, 2023
486db96
docs: add retile to description
amfage Jul 19, 2023
608deed
fix: semaphone indentation
amfage Jul 19, 2023
35a4a8a
fix: standardising workflowtemplate name and container versions
amfage Jul 20, 2023
3cfcc33
fix: add include to tile-index-validate step
amfage Jul 21, 2023
fd8d93f
fix: standardise publish workflow update
MDavidson17 Jul 25, 2023
4a868f6
feat: group use artifact output files
paulfouquet Jul 26, 2023
75b206f
docs: update README
paulfouquet Jul 30, 2023
cde64db
docs: fix typo
paulfouquet Jul 30, 2023
d421768
docs: fix typo
paulfouquet Jul 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 25 additions & 8 deletions templates/argo-tasks/group.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,31 +10,48 @@ spec:
- name: main
inputs:
artifacts:
#
# JSON array of things to group
#
# @example
# ```json
# ["a.json", "b.json", "c.json"]
# ```
- name: input
path: /tmp/group/input.json
optional: true

parameters:
- name: size
description: group into this number of records per group
description: Group into this number of records per group

- name: version
description: container version to use
description: argo-task Container version to use
default: "v2"

outputs:
parameters:
# Grouped output of the input
#
# JSON array of all the group ids, which correspond to a output artifact file
#
# @example
# ```json
# ["000", "001", "002", "003"]
# ```
#
# Workflows should use the "000" string to access the data from the output artifact folder
- name: output
valueFrom:
path: /tmp/group/output.json

artifacts:
# Grouped output of the input
# Grouped output of the input as one file per output groupId
#
# - /output/000.json
# - /output/001.json
# - ...
#
- name: output
path: /tmp/group/output.json
archive:
none: {}
path: /tmp/group/output/

container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/eks:argo-tasks-{{= inputs.parameters.version }}"
Expand Down
12 changes: 11 additions & 1 deletion templates/argo-tasks/tile-index-validate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ spec:
inputs:
parameters:
- name: scale
description: Tile Index scale to validate against
description: Tile Index scale to validate against or target scale for retiling

- name: source_epsg
description: Force source EPSG, "" will attempt to load EPSG from the source tiffs
Expand All @@ -25,6 +25,14 @@ spec:
description: Limit the input list to this regexp
default: ""

- name: validate
description: Validate that all input tiffs perfectly align to tile grid
default: "true"

- name: retile
description: Output tile configuration for retiling
default: "false"

- name: version
description: container version to use
default: "v2"
Expand Down Expand Up @@ -62,6 +70,8 @@ spec:
- "tileindex-validate"
- "--verbose"
- "--scale={{= inputs.parameters.scale }}"
- "--validate={{= inputs.parameters.validate }}"
- "--retile={{= inputs.parameters.retile }}"
- "{{= sprig.empty(inputs.parameters.source_epsg) ? '' : '--source-epsg=' + inputs.parameters.source_epsg }}"
- "{{= sprig.empty(inputs.parameters.include) ? '' : '--include=' + inputs.parameters.include }}"
- "{{= sprig.trim(inputs.parameters.source) }}"
8 changes: 6 additions & 2 deletions workflows/imagery/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ In addition, a Basemaps link is produced enabling visual QA.
| source | str | s3://linz-imagery-staging/test/sample | the uri (path) to the input tiffs |
| include | regex | .tiff?$ | A regular expression to match object path(s) or name(s) from within the source path to include in standardising\*. |
| scale | enum | 500 | The scale of the TIFFs |
| validate | enum | true | Validate the TIFFs files with `tileindex-validate`. |
| retile | enum | false | Prepare the data for retiling TIFFs files to `scale` with `tileindex-validate`. |
| group | int | 50 | The number of files to group into the pods (testing has recommended using 50 for large datasets). |
| compression | enum | webp | Standardised file format |
| cutline | str | | (Optional) location of a cutline file to cut the imagery to `.fgb` or `.geojson` (leave blank if no cutline) |
Expand All @@ -31,8 +33,8 @@ In addition, a Basemaps link is produced enabling visual QA.
| start-datetime | str | YYYY-MM-DD | Imagery start date (flown from), must be in default formatting |
| end-datetime | str | YYYY-MM-DD | Imagery end date (flown to), must be in default formatting |
| copy-option | enum | --no-clobber | `--no-clobber` Skip overwriting existing files. `--force` Overwrite all files. `--force-no-clobber` Overwrite only changed files, skip unchanged files. |
| source-espg | str | 2193 | The EPSP code of the source imagery |
| target-espg | str | 2193 | The target EPSP code - if different to source the imagery will be reprojected |
| source-epsg | str | 2193 | The EPSG code of the source imagery |
| target-epsg | str | 2193 | The target EPSG code - if different to source the imagery will be reprojected |

\* This regex can be used to exclude paths as well, e.g. if there are RBG and RGBI directories, the following regex will only include TIFF files in the RGB directory: `RGB(?!I).*.tiff?$`. For more complicated exclusions, there is an `--exclude` parameter, which would need to be added to the Argo WorkflowTemplate.

Expand All @@ -43,6 +45,8 @@ In addition, a Basemaps link is produced enabling visual QA.
| source | s3://linz-imagery-upload/PRJ39741_BOPLASS_Imagery_2021-22/PRJ39741_03/01_GeoTiff/ |
| include | .tiff?$ |
| scale | 2000 |
| validate | true |
| retile | false |
| group | 50 |
| compression | webp |
| cutline | s3://linz-imagery-staging/cutline/bay-of-plenty_2021-2022.fgb |
Expand Down
4 changes: 4 additions & 0 deletions workflows/imagery/standardising-publish-import.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ spec:
value: ".tiff?$"
- name: transform
value: "f"
- name: validate
value: "true"
- name: retile
value: "false"
- name: version-argo-tasks
value: "v2.9"
- name: version-basemaps-cli
Expand Down
Loading
Loading