Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: switch to npm and improve formatting #192

Merged
merged 5 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module.exports = {
...require('@linzjs/style/.eslintrc.cjs'),
};
12 changes: 6 additions & 6 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,21 @@ jobs:
- name: Use Node.js 18.x
uses: actions/setup-node@v2.2.0
with:
node-version: "18.x"
node-version: '18.x'

- name: Install
run: yarn
run: npm install --ci

- name: format
run: yarn format
run: npm run lint -- --fix=false # ensure eslint is not configured to --fix

- name: Check formatting
run: git diff --exit-code
- name: format
run: npm run format -- --fix=false # ensure eslint is not configured to --fix

- name: Setup kubectl
uses: azure/setup-kubectl@v3
with:
version: "latest"
version: 'latest'

- name: AWS Configure
uses: aws-actions/configure-aws-credentials@v1.6.1
Expand Down
3 changes: 3 additions & 0 deletions .prettierrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module.exports = {
...require('@linzjs/style/.prettierrc.cjs'),
};
17 changes: 6 additions & 11 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,7 @@ container:
env:
- name: AWS_ROLE_CONFIG_PATH
value: s3://linz-bucket-config/config.json
args:
[
"stac-validate",
"--recursive",
"{{inputs.parameters.location}}flat/collection.json",
]
args: ['stac-validate', '--recursive', '{{inputs.parameters.location}}flat/collection.json']
```

## <a name='WorkflowPerformanceSettings'></a>Workflow Performance Settings
Expand Down Expand Up @@ -122,9 +117,9 @@ Two example workflow tasks requesting specific resources (memory and CPU) from t
```yaml
- name: standardise-validate
retryStrategy:
limit: "2"
limit: '2'
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
inputs:
parameters:
- name: file
Expand All @@ -138,13 +133,13 @@ Two example workflow tasks requesting specific resources (memory and CPU) from t
ephemeral-storage: 3Gi
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
mountPath: '/tmp'
```

```yaml
- name: copy
retryStrategy:
limit: "2"
limit: '2'
inputs:
parameters:
- name: file
Expand Down Expand Up @@ -173,7 +168,7 @@ _Note: the Argo Workflows flowchart UI refers to pods as “nodes”._
In the standardising workflow, the `aws-list` task creates an output of a list of lists. Each `standardise-validate` task pod receives a list of files to process as an input.

```yaml
withParam: "{{tasks.aws-list.outputs.parameters.files}}"
withParam: '{{tasks.aws-list.outputs.parameters.files}}'
```

For example, `aws-list` output with a `group` size of 4 set:
Expand Down
8 changes: 4 additions & 4 deletions config/semaphores.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ metadata:
name: semaphores
namespace: argo
data:
standardising: "2" # Limit of how many standardising workflow instances can run at the same time
bulk: "4" # Limit of how many bulk workflow instances can run at the same time
bulkcopy: "8" # Limit of how many publish-copy workflow instances can run at the same time
basemaps_import: "10" # Limit of how many basemaps import workflow instances can run at the same time
standardising: '2' # Limit of how many standardising workflow instances can run at the same time
bulk: '4' # Limit of how many bulk workflow instances can run at the same time
bulkcopy: '8' # Limit of how many publish-copy workflow instances can run at the same time
basemaps_import: '10' # Limit of how many basemaps import workflow instances can run at the same time
2 changes: 1 addition & 1 deletion docs/concurrency.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ metadata:
name: semaphores
namespace: argo
data:
standardising: "2"
standardising: '2'
```

which shows a limit of `2` added to the key `standardising`. This can then
Expand Down
46 changes: 23 additions & 23 deletions docs/training_workshop/ARGO_WORKFLOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ metadata:
namespace: argo
spec:
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
entrypoint: main
templates:
- name: main
Expand All @@ -129,9 +129,9 @@ spec:
template: say-hello-template
- name: say-hello-template
container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3"
image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3'
command: [echo]
args: ["hello world"]
args: ['hello world']
```

Example workflow file: [wf_hello_world.yaml](example_workflows/wf_hello_world.yaml)
Expand All @@ -142,7 +142,7 @@ You will see the following specified in the example workflows for this workshop.

```yaml
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
```

Note: prefixing the name of the workflow with `test-` prevents alerts for the workflow being sent to the `#alert-argo-workflows` Slack channel.
Expand Down Expand Up @@ -174,7 +174,7 @@ spec:
arguments:
parameters:
- name: message
value: "hello world"
value: 'hello world'
```

The parameter can be referenced in the workflow using Argo variables as `"{{workflow.parameters.message}}"`
Expand Down Expand Up @@ -269,17 +269,17 @@ argo submit topo-workflows/imagery/standardising-publish-import.yaml -n argo -f
_params.yaml_:

```yaml
source: "s3://linz-imagery-source-example/aerial-imagery/new-zealand/christchurch_urban_2021_0.05m_RGB/"
target: "s3://linz-imagery-example/canterbury/christchurch_2021_0.05m/rgb/2193/"
scale: "500"
group: "29"
cutline: "s3://linz-imagery-cutline-example/historical-imagery-cutlines/2023-01-16_84fd68f/SNC50451-combined.fgb"
title: "Christchurch 0.05m Urban Aerial Photos (2021)"
description: "Orthophotography within the Canterbury region captured in the 2021 flying season."
producer: "Aerial Surveys"
licensor: "Toitū Te Whenua Land Information New Zealand"
start-datetime: "2021-11-02"
end-datetime: "2021-12-02"
source: 's3://linz-imagery-source-example/aerial-imagery/new-zealand/christchurch_urban_2021_0.05m_RGB/'
target: 's3://linz-imagery-example/canterbury/christchurch_2021_0.05m/rgb/2193/'
scale: '500'
group: '29'
cutline: 's3://linz-imagery-cutline-example/historical-imagery-cutlines/2023-01-16_84fd68f/SNC50451-combined.fgb'
title: 'Christchurch 0.05m Urban Aerial Photos (2021)'
description: 'Orthophotography within the Canterbury region captured in the 2021 flying season.'
producer: 'Aerial Surveys'
licensor: 'Toitū Te Whenua Land Information New Zealand'
start-datetime: '2021-11-02'
end-datetime: '2021-12-02'
```

> **_Resources:_** [Argo CLI documentation](https://argoproj.github.io/argo-workflows/cli/argo/) - [Kubernetes Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)
Expand Down Expand Up @@ -343,14 +343,14 @@ metadata:
namespace: argo
spec:
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
entrypoint: main
arguments:
parameters:
- name: message1
value: "hello world 1"
value: 'hello world 1'
- name: message2
value: "hello world 2"
value: 'hello world 2'
templates:
- name: main
dag:
Expand All @@ -360,21 +360,21 @@ spec:
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message1}}"
value: '{{workflow.parameters.message1}}'
- name: say-hello-task2
template: say-hello-template
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message2}}"
value: '{{workflow.parameters.message2}}'
- name: say-hello-template
inputs:
parameters:
- name: message
container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3"
image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3'
command: [echo]
args: ["{{inputs.parameters.message}}"]
args: ['{{inputs.parameters.message}}']
```

Example workflow file: [wf_hello_world_args_tasks.yaml](example_workflows/wf_hello_world_args_tasks.yaml)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metadata:
namespace: argo
spec:
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
entrypoint: main
templates:
- name: main
Expand All @@ -16,6 +16,6 @@ spec:
template: say-hello-template
- name: say-hello-template
container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3"
image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3'
command: [echo]
args: ["hello world"]
args: ['hello world']
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ metadata:
namespace: argo
spec:
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
entrypoint: main
arguments:
parameters:
- name: message
value: "hello world"
value: 'hello world'
templates:
- name: main
dag:
Expand All @@ -21,12 +21,12 @@ spec:
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message}}"
value: '{{workflow.parameters.message}}'
- name: say-hello-template
inputs:
parameters:
- name: message
container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3"
image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3'
command: [echo]
args: ["{{inputs.parameters.message}}"]
args: ['{{inputs.parameters.message}}']
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ metadata:
namespace: argo
spec:
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
entrypoint: main
arguments:
parameters:
- name: message1
value: "hello world 1"
value: 'hello world 1'
- name: message2
value: "hello world 2"
value: 'hello world 2'
templates:
- name: main
dag:
Expand All @@ -23,18 +23,18 @@ spec:
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message1}}"
value: '{{workflow.parameters.message1}}'
- name: say-hello-task2
template: say-hello-template
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message2}}"
value: '{{workflow.parameters.message2}}'
- name: say-hello-template
inputs:
parameters:
- name: message
container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3"
image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3'
command: [echo]
args: ["{{inputs.parameters.message}}"]
args: ['{{inputs.parameters.message}}']
24 changes: 12 additions & 12 deletions docs/training_workshop/example_workflows/wf_hello_world_dag.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ metadata:
namespace: argo
spec:
nodeSelector:
karpenter.sh/capacity-type: "spot"
karpenter.sh/capacity-type: 'spot'
entrypoint: main
arguments:
parameters:
- name: message1
value: "hello world 1"
value: 'hello world 1'
- name: message2
value: "hello world 2"
value: 'hello world 2'
templates:
- name: main
dag:
Expand All @@ -23,33 +23,33 @@ spec:
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message1}}"
value: '{{workflow.parameters.message1}}'
- name: say-hello-task2
template: say-hello-template
arguments:
parameters:
- name: message
value: "{{workflow.parameters.message2}}"
depends: "say-hello-task1"
value: '{{workflow.parameters.message2}}'
depends: 'say-hello-task1'
- name: say-hello-task3
template: say-hello-template
arguments:
parameters:
- name: message
value: "say hello task 3"
depends: "say-hello-task1"
value: 'say hello task 3'
depends: 'say-hello-task1'
- name: say-hello-task4
template: say-hello-template
arguments:
parameters:
- name: message
value: "say hello task 4"
depends: "say-hello-task2 && say-hello-task3"
value: 'say hello task 4'
depends: 'say-hello-task2 && say-hello-task3'
- name: say-hello-template
inputs:
parameters:
- name: message
container:
image: "019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3"
image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:v3'
command: [echo]
args: ["{{inputs.parameters.message}}"]
args: ['{{inputs.parameters.message}}']
Loading
Loading