Skip to content

Commit

Permalink
docs(airflow): add known limitations for automatic lineage (datahub-p…
Browse files Browse the repository at this point in the history 
…roject#11652)
  • Loading branch information
@hsheth2
hsheth2 authored Oct 17, 2024
1 parent 523a456 commit d7ccf3b
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion docs/lineage/airflow.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ conn_id = datahub_rest_default # or datahub_kafka_default
```

| Name | Default value | Description |
|----------------------------|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| -------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| enabled | true | If the plugin should be enabled. |
| conn_id | datahub_rest_default | The name of the datahub connection you set in step 1. |
| cluster | prod | name of the airflow cluster |
Expand Down Expand Up @@ -191,6 +191,10 @@ These operators are supported by OpenLineage, but we haven't tested them yet:
There's also a few operators (e.g. BashOperator, PythonOperator) that have custom extractors, but those extractors don't generate lineage.
-->

Known limitations:

- We do not fully support operators that run multiple SQL statements at once. In these cases, we'll only capture lineage from the first SQL statement.

## Manual Lineage Annotation

### Using `inlets` and `outlets`
Expand Down

0 comments on commit d7ccf3b

Please sign in to comment.