Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: create national DEMs workflow TDE-1166 #792

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

feat: create national DEMs workflow TDE-1166 #792

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bce334d
feat: create national DEMs workflow TDE-1166
paulfouquet Oct 6, 2024
4de806b
fix: capture-dates.geojson path is wrong
paulfouquet Oct 7, 2024
4dac226
fix: create-collection parameters are wrong
paulfouquet Oct 8, 2024
1593ff2
feat: national dem path without dates
paulfouquet Oct 10, 2024
95b9461
fix: collection.json should link capture-dates.geojson
paulfouquet Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions workflows/raster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
- [Standardising](#Standardising)
- [copy](#copy)
- [publish-odr](#Publish-odr)
- [National DEM](#national-dem)
- [tests](#Tests)

# Standardising
Expand Down Expand Up @@ -280,6 +281,35 @@ graph TD;

See the [copy template](#copy) for more information.

# national-dem

This workflow combines a set of DEMs datasets in order to create a single national dataset composed of 50k tiles.

Upon completion all standardised TIFF and STAC files will be located with the ./flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.

Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#Publish-odr) that can be run automatically after standardisation.

## Workflow Input Parameters

| Parameter | Type | Default | Description |
| ------------------ | ---- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| config-file | str | https://raw.githubusercontent.com/linz/basemaps-config/master/config/tileset/elevation.json | Location of the configuration file listing the source datasets to merge. |
| compare | str | | Existing collection.json to compare with the new merging. |
| source_epsg | str | 2193 | The EPSG code of the source imagery |
| target_epsg | str | 2193 | The target EPSG code - if different to source the imagery will be reprojected |
| group | 4 | | How many output tiles to process in each standardising task "pod". Change if you have resource or performance issues when standardising a dataset. |
| collection_id | 4 | | Collection ID of the existing National DEM collection. |
| publish_to_odr | str | false | Run [publish-odr](#Publish-odr) after standardising has completed successfully |
| target_bucket_name | enum | | Used only if `publish_to_odr` is true. The bucket name of the target ODR location |
| copy_option | enum | --no-clobber | Used only if `publish_to_odr` is true.<dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |

### Example Input Parameters

| Parameter | Value |
| ------------- | --------------------------------------------------------------------- |
| compare | s3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/collection.json |
| collection_id | 01J6TK9HHNDHJEG8QRSF98WH11 |

# Tests

## How To Use the Test Workflow
Expand Down
Loading