Skip to content

Commit

Permalink
feat: tool to generate elevation argo commands TDE-783 (#144)
Browse files Browse the repository at this point in the history
sample output parameters yaml:
```
"source": "s3://linz-elevation-staging/lidar2/Auckland_South_LiDAR_(2016-2017)/Auckland_South_LiDAR_1m_DSM_(2016-2017)/"
"target": "s3://linz-elevation/auckland/auckland-south_2016-2017/dsm_1m/2193/"
"title": "Auckland South LiDAR 1m DSM (2016-2017)"
"description": "Digital Surface Model within the Auckland South region captured in 2016-2017."
"start-datetime": "2016-09-22"
"end-datetime": "2017-06-22"
"scale": "10000"
"source-epsg": "2193"
"target-epsg": "2193"
"compression": "dem_lerc"
"retile": "true"
"validate": "false"
"group": "1"
"licensor": "Auckland Council"
"licensor-list": ""
"producer": "AAM NZ"
"producer-list": ""
```

sample output submit script:
```
#!/bin/bash

argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./mackenzie-2015-dem-1m.yaml --generate-name mackenzie-2015-dem-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./mackenzie-2015-dsm-1m.yaml --generate-name mackenzie-2015-dsm-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./queenstown-2016-dem-1m.yaml --generate-name queenstown-2016-dem-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./queenstown-2016-dsm-1m.yaml --generate-name queenstown-2016-dsm-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./blenheim-2014-dem-1m.yaml --generate-name blenheim-2014-dem-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./blenheim-2014-dsm-1m.yaml --generate-name blenheim-2014-dsm-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./reporoa-and-upper-piako-river-2019-dem-1m.yaml --generate-name reporoa-and-upper-piako-river-2019-dem-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./reporoa-and-upper-piako-river-2019-dsm-1m.yaml --generate-name reporoa-and-upper-piako-river-2019-dsm-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./otago-2016-dem-1m.yaml --generate-name otago-2016-dem-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./otago-2016-dsm-1m.yaml --generate-name otago-2016-dsm-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./motueka-river-valley-2018-2019-dem-1m.yaml --generate-name motueka-river-valley-2018-2019-dem-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./motueka-river-valley-2018-2019-dsm-1m.yaml --generate-name motueka-river-valley-2018-2019-dsm-1m
argo submit ~/dev/topo-workflows/workflows/imagery/standardising-publish-import.yaml -n argo -f ./whangarei-heads-2016-dem-1m.yaml --generate-name whangarei-heads-2016-dem-1m

# tauranga-and-coast-2015-dem-1m.yaml not written to bash as further action required: ASC FILES
# tauranga-and-coast-2015-dsm-1m.yaml not written to bash as further action required: ASC FILES
# west-coast-and-hauraki-plains-2015-dem-1m.yaml not written to bash as further action required: ASC FILES
# west-coast-and-hauraki-plains-2015-dsm-1m.yaml not written to bash as further action required: ASC FILES

```
  • Loading branch information
MDavidson17 authored Aug 10, 2023
1 parent 04675a8 commit b3703d6
Show file tree
Hide file tree
Showing 3 changed files with 287 additions and 0 deletions.
34 changes: 34 additions & 0 deletions tools/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Tools

This folder contains single use scripts which have been used to assist in automating argo jobs.
The scripts should be stored in this folder if they may become useful again in the future.

## generate-argo-cli-commands-elevation.py

**Date:** 31/07/2023

**Related Jira Tickets:** [TDE-783](https://toitutewhenua.atlassian.net/browse/TDE-783?atlOrigin=eyJpIjoiODhlMWFkYTVlYzhiNGJjYTgwOTI4OWVjNmQ3YzczMjciLCJwIjoiaiJ9)

**Description:**
This script sets up for the automated processing of numerous elevation datasets using the argo cli.

**Instructions:**

1. Run:

```bash
cd ./tools
python3 generate-argo-cli-commands-elevation.py
```

**Output:**

- **region-year-datatype-scale.yaml:** workflow parameters for this dataset
- **standardise-publish.sh:** bash script to 'deploy' argo workflows
**nb: the commented lines at the end of this file detail the datasets not run due to know issues.**

**Submitting:**

```bash
sh standardise-publish.sh
```
Loading

0 comments on commit b3703d6

Please sign in to comment.