Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix] Save Pipeline resource to state in addition to spec #3869

Merged
merged 7 commits into from
Aug 14, 2024
Merged

[Fix] Save Pipeline resource to state in addition to spec #3869

merged 7 commits into from
Aug 14, 2024

Conversation

mgyucht
Copy link
Contributor

@mgyucht mgyucht commented Aug 8, 2024 •
edited by alexott
Loading

Changes

After #3839, the provider's behavior of DLT pipelines regressed. In particular, the Read method stopped populating certain fields from the GetPipelineResponse into Terraform state. This PR addresses this by additionally writing all top-level fields into the state as part of the read operation.

Resolves #3855.

Tests

Unit tests cover the case specified in the issue.

  • make test run locally
  • relevant change in docs/ folder
  • covered with integration tests in internal/acceptance
  • relevant acceptance tests are passing
  • using Go SDK

@mgyucht mgyucht requested review from a team as code owners August 8, 2024 11:42
@mgyucht mgyucht requested review from tanmay-db and removed request for a team August 8, 2024 11:42
@mgyucht mgyucht changed the title Save Pipeline resource to state in addition to spec [Fix] Save Pipeline resource to state in addition to spec Aug 8, 2024
@mgyucht mgyucht added this pull request to the merge queue Aug 14, 2024
@mgyucht mgyucht removed this pull request from the merge queue due to a manual request Aug 14, 2024
@mgyucht mgyucht enabled auto-merge August 14, 2024 09:01
@mgyucht mgyucht added this pull request to the merge queue Aug 14, 2024
Merged via the queue into main with commit 42e1a45 Aug 14, 2024
6 checks passed
@mgyucht mgyucht deleted the issue-3855 branch August 14, 2024 09:06
mgyucht added a commit that referenced this pull request Aug 14, 2024
@mgyucht
[Release] Release v1.49.2
f235ec9 
### New Features and Improvements

 * Added support for `cloudflare_api_token` in `databricks_storage_credential` resource ([#3835](#3835)).
 * Add `active` attribute to `databricks_user` data source ([#3733](#3733)).
 * Add `workspace_path` attribute to `databricks_notebook` resource and data source ([#3885](#3885)).
 * Mark attributes as sensitive in `databricks_mlflow_webhook` ([#3825](#3825)).
 * Added notification destination resource ([#3820](#3820)).

### Bug Fixes

 * Automatically assign `IS_OWNER` permission to sql warehouse if not specified ([#3829](#3829)).
 * Corrected kms arn format in `data_aws_unity_catalog_policy` ([#3823](#3823)).
 * Fix crash when destroying `databricks_compliance_security_profile_workspace_setting` ([#3883](#3883)).
 * Fixed read method of `databricks_entitlements` resource ([#3858](#3858)).
 * Retry cluster update on "INVALID_STATE" ([#3890](#3890)).
 * Save Pipeline resource to state in addition to spec ([#3869](#3869)).
 * Tolerate `databricks_workspace_conf` deletion failures ([#3737](#3737)).
 * Update Go SDK ([#3826](#3826)).
 * cluster key update for `databricks_sql_table` should not force new ([#3824](#3824)).
 * reading `databricks_metastore_assignment` when importing resource ([#3827](#3827)).

### Documentation

 * Add troubleshooting instructions for `databricks OAuth is not supported for this host` error ([#3815](#3815)).
 * Clarify setting of permissions for workspace objects ([#3884](#3884)).
 * Document missing task attributes in `databricks_job` resource ([#3817](#3817)).
 * Fixed documentation for `databricks_schemas` data source and `databricks_metastore_assignment` resource ([#3851](#3851)).
 * clarified `spot_bid_max_price` option for `databricks_cluster` ([#3830](#3830)).
 * marked `databricks_sql_dashboard` as legacy ([#3836](#3836)).

### Internal Changes

 * Refactor exporter: split huge files into smaller ones ([#3870](#3870)).
 * Refactored `client.ClientForHost` to use Go SDK method ([#3735](#3735)).
 * Revert "Rewriting DLT pipelines using SDK" ([#3838](#3838)).
 * Rewrite DLT pipelines using SDK ([#3839](#3839)).
 * Rewriting DLT pipelines using SDK ([#3792](#3792)).
 * Update Go SDK ([#3808](#3808)).
 * refactored `databricks_mws_permission_assignment` to Go SDK ([#3831](#3831)).

### Dependency Updates

 * Bump databricks-sdk-go to 0.44.0 ([#3896](#3896)).
 * Bump github.com/zclconf/go-cty from 1.14.4 to 1.15.0 ([#3775](#3775)).

### Exporter

 * Add retry on "Operation timed out" error ([#3897](#3897)).
 * Add support for Vector Search assets ([#3828](#3828)).
 * Add support for `databricks_notification_destination` ([#3861](#3861)).
 * Add support for `databricks_online_table` ([#3816](#3816)).
 * Don't export model serving endpoints with foundational models ([#3845](#3845)).
 * Fix generation of `autotermination_minutes = 0` ([#3881](#3881)).
 * Generate `databricks_workspace_binding` instead of legacy `databricks_catalog_workspace_binding` ([#3812](#3812)).
 * Ignore DLT pipelines deployed via DABs ([#3857](#3857)).
 * Improve exporting of `databricks_model_serving` ([#3821](#3821)).
 * Refactoring: remove legacy code ([#3864](#3864)).
mgyucht added a commit that referenced this pull request Aug 14, 2024
@mgyucht
[Release] Release v1.50.0
05af14d 
### New Features and Improvements

 * Added `databricks_notification_destination` resource ([#3820](#3820)).
 * Added support for `cloudflare_api_token` in `databricks_storage_credential` resource ([#3835](#3835)).
 * Add `active` attribute to `databricks_user` data source ([#3733](#3733)).
 * Add `workspace_path` attribute to `databricks_notebook` resource and data source ([#3885](#3885)).
 * Mark attributes as sensitive in `databricks_mlflow_webhook` ([#3825](#3825)).

### Bug Fixes

 * Automatically assign `IS_OWNER` permission to sql warehouse if not specified ([#3829](#3829)).
 * Corrected kms arn format in `data_aws_unity_catalog_policy` ([#3823](#3823)).
 * Fix crash when destroying `databricks_compliance_security_profile_workspace_setting` ([#3883](#3883)).
 * Fixed read method of `databricks_entitlements` resource ([#3858](#3858)).
 * Retry cluster update on "INVALID_STATE" ([#3890](#3890)).
 * Save Pipeline resource to state in addition to spec ([#3869](#3869)).
 * Tolerate `databricks_workspace_conf` deletion failures ([#3737](#3737)).
 * Update Go SDK ([#3826](#3826)).
 * cluster key update for `databricks_sql_table` should not force new ([#3824](#3824)).
 * reading `databricks_metastore_assignment` when importing resource ([#3827](#3827)).

### Documentation

 * Add troubleshooting instructions for `databricks OAuth is not supported for this host` error ([#3815](#3815)).
 * Clarify setting of permissions for workspace objects ([#3884](#3884)).
 * Document missing task attributes in `databricks_job` resource ([#3817](#3817)).
 * Fixed documentation for `databricks_schemas` data source and `databricks_metastore_assignment` resource ([#3851](#3851)).
 * clarified `spot_bid_max_price` option for `databricks_cluster` ([#3830](#3830)).
 * marked `databricks_sql_dashboard` as legacy ([#3836](#3836)).

### Internal Changes

 * Refactor exporter: split huge files into smaller ones ([#3870](#3870)).
 * Refactored `client.ClientForHost` to use Go SDK method ([#3735](#3735)).
 * Revert "Rewriting DLT pipelines using SDK" ([#3838](#3838)).
 * Rewrite DLT pipelines using SDK ([#3839](#3839)).
 * Rewriting DLT pipelines using SDK ([#3792](#3792)).
 * Update Go SDK ([#3808](#3808)).
 * refactored `databricks_mws_permission_assignment` to Go SDK ([#3831](#3831)).

### Dependency Updates

 * Bump databricks-sdk-go to 0.44.0 ([#3896](#3896)).
 * Bump github.com/zclconf/go-cty from 1.14.4 to 1.15.0 ([#3775](#3775)).

### Exporter

 * Add retry on "Operation timed out" error ([#3897](#3897)).
 * Add support for Vector Search assets ([#3828](#3828)).
 * Add support for `databricks_notification_destination` ([#3861](#3861)).
 * Add support for `databricks_online_table` ([#3816](#3816)).
 * Don't export model serving endpoints with foundational models ([#3845](#3845)).
 * Fix generation of `autotermination_minutes = 0` ([#3881](#3881)).
 * Generate `databricks_workspace_binding` instead of legacy `databricks_catalog_workspace_binding` ([#3812](#3812)).
 * Ignore DLT pipelines deployed via DABs ([#3857](#3857)).
 * Improve exporting of `databricks_model_serving` ([#3821](#3821)).
 * Refactoring: remove legacy code ([#3864](#3864)).
@mgyucht mgyucht mentioned this pull request Aug 14, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Aug 15, 2024
@mgyucht
[Release] Release v1.50.0 (#3900)
79b9377 
### New Features and Improvements

* Added `databricks_notification_destination` resource
([#3820](#3820)).
* Added support for `cloudflare_api_token` in
`databricks_storage_credential` resource
([#3835](#3835)).
* Add `active` attribute to `databricks_user` data source
([#3733](#3733)).
* Add `workspace_path` attribute to `databricks_notebook` resource and
data source
([#3885](#3885)).
* Mark attributes as sensitive in `databricks_mlflow_webhook`
([#3825](#3825)).


### Bug Fixes

* Automatically assign `IS_OWNER` permission to sql warehouse if not
specified
([#3829](#3829)).
* Corrected kms arn format in `data_aws_unity_catalog_policy`
([#3823](#3823)).
* Fix crash when destroying
`databricks_compliance_security_profile_workspace_setting`
([#3883](#3883)).
* Fixed read method of `databricks_entitlements` resource
([#3858](#3858)).
* Retry cluster update on "INVALID_STATE"
([#3890](#3890)).
* Save Pipeline resource to state in addition to spec
([#3869](#3869)).
* Tolerate `databricks_workspace_conf` deletion failures
([#3737](#3737)).
* Update Go SDK
([#3826](#3826)).
* cluster key update for `databricks_sql_table` should not force new
([#3824](#3824)).
* reading `databricks_metastore_assignment` when importing resource
([#3827](#3827)).


### Documentation

* Add troubleshooting instructions for `databricks OAuth is not
supported for this host` error
([#3815](#3815)).
* Clarify setting of permissions for workspace objects
([#3884](#3884)).
* Document missing task attributes in `databricks_job` resource
([#3817](#3817)).
* Fixed documentation for `databricks_schemas` data source and
`databricks_metastore_assignment` resource
([#3851](#3851)).
* clarified `spot_bid_max_price` option for `databricks_cluster`
([#3830](#3830)).
* marked `databricks_sql_dashboard` as legacy
([#3836](#3836)).


### Internal Changes

* Refactor exporter: split huge files into smaller ones
([#3870](#3870)).
* Refactored `client.ClientForHost` to use Go SDK method
([#3735](#3735)).
* Revert "Rewriting DLT pipelines using SDK"
([#3838](#3838)).
* Rewrite DLT pipelines using SDK
([#3839](#3839)).
* Rewriting DLT pipelines using SDK
([#3792](#3792)).
* Update Go SDK
([#3808](#3808)).
* refactored `databricks_mws_permission_assignment` to Go SDK
([#3831](#3831)).


### Dependency Updates

* Bump databricks-sdk-go to 0.44.0
([#3896](#3896)).
* Bump github.com/zclconf/go-cty from 1.14.4 to 1.15.0
([#3775](#3775)).


### Exporter

* Add retry on "Operation timed out" error
([#3897](#3897)).
* Add support for Vector Search assets
([#3828](#3828)).
* Add support for `databricks_notification_destination`
([#3861](#3861)).
* Add support for `databricks_online_table`
([#3816](#3816)).
* Don't export model serving endpoints with foundational models
([#3845](#3845)).
* Fix generation of `autotermination_minutes = 0`
([#3881](#3881)).
* Generate `databricks_workspace_binding` instead of legacy
`databricks_catalog_workspace_binding`
([#3812](#3812)).
* Ignore DLT pipelines deployed via DABs
([#3857](#3857)).
* Improve exporting of `databricks_model_serving`
([#3821](#3821)).
* Refactoring: remove legacy code
([#3864](#3864)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexott alexott alexott approved these changes

@tanmay-db tanmay-db Awaiting requested review from tanmay-db tanmay-db is a code owner automatically assigned from databricks/eng-plat-auto-exp-reviewers

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[ISSUE] Issue with databricks_pipeline resource after migrating it to Go SDK
2 participants
@mgyucht @alexott