Skip to content

Commit

Permalink
[Doc] Document missing task attributes in databricks_job resource (#…
Browse files Browse the repository at this point in the history
…3817)

## Changes
<!-- Summary of your changes that are easy to understand -->

- Added documentation for `description` and `disable_auto_optimization`
attributes.
- Sorted task attributes by name for easier search

## Tests
<!-- 
How is this tested? Please see the checklist below and also describe any
other relevant tests
-->

- [ ] `make test` run locally
- [x] relevant change in `docs/` folder
- [ ] covered with integration tests in `internal/acceptance`
- [ ] relevant acceptance tests are passing
- [ ] using Go SDK
  • Loading branch information
alexott authored Jul 26, 2024
1 parent 01be651 commit 0111483
Showing 1 changed file with 10 additions and 8 deletions.
18 changes: 10 additions & 8 deletions docs/resources/job.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,20 +125,22 @@ This block describes individual tasks:
* `spark_python_task`
* `spark_submit_task`
* `sql_task`
* `library` - (Optional) (Set) An optional list of libraries to be installed on the cluster that will execute the job.
* `depends_on` - (Optional) block specifying dependency(-ies) for a given task.
* `job_cluster_key` - (Optional) Identifier of the Job cluster specified in the `job_cluster` block.
* `description` - (Optional) description for this task.
* `disable_auto_optimization` - (Optional) A flag to disable auto optimization in serverless tasks.
* `email_notifications` - (Optional) An optional block to specify a set of email addresses notified when this task begins, completes or fails. The default behavior is to not send any emails. This block is [documented below](#email_notifications-configuration-block).
* `environment_key` - (Optional) identifier of an `environment` block that is used to specify libraries. Required for some tasks (`spark_python_task`, `python_wheel_task`, ...) running on serverless compute.
* `existing_cluster_id` - (Optional) Identifier of the [interactive cluster](cluster.md) to run job on. *Note: running tasks on interactive clusters may lead to increased costs!*
* `health` - (Optional) block described below that specifies health conditions for a given task.
* `job_cluster_key` - (Optional) Identifier of the Job cluster specified in the `job_cluster` block.
* `library` - (Optional) (Set) An optional list of libraries to be installed on the cluster that will execute the job.
* `max_retries` - (Optional) (Integer) An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with a `FAILED` or `INTERNAL_ERROR` lifecycle state. The value -1 means to retry indefinitely and the value 0 means to never retry. The default behavior is to never retry. A run can have the following lifecycle state: `PENDING`, `RUNNING`, `TERMINATING`, `TERMINATED`, `SKIPPED` or `INTERNAL_ERROR`.
* `min_retry_interval_millis` - (Optional) (Integer) An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
* `new_cluster` - (Optional) Task will run on a dedicated cluster. See [databricks_cluster](cluster.md) documentation for specification. *Some parameters, such as `autotermination_minutes`, `is_pinned`, `workload_type` aren't supported!*
* `environment_key` - (Optional) identifier of an `environment` block that is used to specify libraries. Required for some tasks (`spark_python_task`, `python_wheel_task`, ...) running on serverless compute.
* `run_if` - (Optional) An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. One of `ALL_SUCCESS`, `AT_LEAST_ONE_SUCCESS`, `NONE_FAILED`, `ALL_DONE`, `AT_LEAST_ONE_FAILED` or `ALL_FAILED`. When omitted, defaults to `ALL_SUCCESS`.
* `retry_on_timeout` - (Optional) (Bool) An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
* `max_retries` - (Optional) (Integer) An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with a `FAILED` or `INTERNAL_ERROR` lifecycle state. The value -1 means to retry indefinitely and the value 0 means to never retry. The default behavior is to never retry. A run can have the following lifecycle state: `PENDING`, `RUNNING`, `TERMINATING`, `TERMINATED`, `SKIPPED` or `INTERNAL_ERROR`.
* `run_if` - (Optional) An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. One of `ALL_SUCCESS`, `AT_LEAST_ONE_SUCCESS`, `NONE_FAILED`, `ALL_DONE`, `AT_LEAST_ONE_FAILED` or `ALL_FAILED`. When omitted, defaults to `ALL_SUCCESS`.
* `timeout_seconds` - (Optional) (Integer) An optional timeout applied to each run of this job. The default behavior is to have no timeout.
* `min_retry_interval_millis` - (Optional) (Integer) An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
* `email_notifications` - (Optional) (List) An optional set of email addresses notified when this task begins, completes or fails. The default behavior is to not send any emails. This field is a block and is [documented below](#email_notifications-configuration-block).
* `webhook_notifications` - (Optional) (List) An optional set of system destinations (for example, webhook destinations or Slack) to be notified when runs of this task begins, completes or fails. The default behavior is to not send any notifications. This field is a block and is documented below.
* `health` - (Optional) block described below that specifies health conditions for a given task.

-> **Note** If no `job_cluster_key`, `existing_cluster_id`, or `new_cluster` were specified in task definition, then task will executed using serverless compute.

Expand Down

0 comments on commit 0111483

Please sign in to comment.