New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Internal] Added `databricks_quality_monitor` resource and `databricks_volumes` data source to plugin framework #3958

Merged

tanmay-db merged 23 commits into main from plugin-fw-resources-datasources

Aug 30, 2024

Contributor

tanmay-db commented Aug 27, 2024 •

edited

Loading

Changes

Add support for Quality Monitor resource and Volumes data source in plugin framework. Also add utility method to get workspace and account client with diagnostics.

Subpart of: #3751 + Addressed comments from: https://github.com/databricks/terraform-provider-databricks/pull/3893/files#diff-8edec0f6289f63bfb61c7c6fb76fc1a61eec69c33fc2ba3a4ee6e77a8103f85b

Note: The changes for flipping databricks_quality_monitor_pluginframework -> databricks_quality_monitor in pluginframework along with making this legacy for sdkv2 i.e. databricks_quality_monitor -> databricks_quality_monitor_legacy will be done in separate PR. Same for databricks_volumes

Tests

Existing unit tests.

Integration and Unit tests for this resource will be merged after this PR is merged.

make test run locally
relevant change in docs/ folder
covered with integration tests in internal/acceptance
relevant acceptance tests are passing
using Go SDK


          [Internal] Added Volumes DataSource and Quality Monitor Resource

dc0b435

tanmay-db requested review from a team as code owners

August 27, 2024 20:56

tanmay-db requested review from mgyucht and removed request for a team and mgyucht

August 27, 2024 20:56

tanmay-db added 3 commits

August 27, 2024 22:59

2af5419

09f6849

c5bbe00

tanmay-db changed the title ~~[Internal] Added Volumes DataSource and Quality Monitor Resource~~ [Internal] Added Volumes DataSource and Quality Monitor Resource to Plugin Framework

tanmay-db changed the title ~~[Internal] Added Volumes DataSource and Quality Monitor Resource to Plugin Framework~~ [Internal] Added databricks_quality_monitor resource and data_volumes data source to plugin framework

tanmay-db changed the title ~~[Internal] Added databricks_quality_monitor resource and data_volumes data source to plugin framework~~ [Internal] Added databricks_quality_monitor resource and databricks_volumes data source to plugin framework

tanmay-db added 3 commits

August 28, 2024 17:24


          comments

4a61953


          Merge branch 'main' of github.com:databricks/terraform-provider-datab…

…ricks into plugin-fw-resources-datasources


          comments

f1543e7

tanmay-db requested review from mgyucht and edwardfeng-db

August 28, 2024 15:28


          comments

ff9db74

mgyucht requested changes

View reviewed changes

Contributor

mgyucht left a comment

Great first start. Overall things seem fine, but I would like to be a bit more convention-oriented w.r.t. how we develop code for the TF provider.

Let's be very consistent with how diag.Diagnostics variables are handled. I think we can eliminate bugs by handling them in exactly two ways:

funcs that return only diag.Diagnostics are called inside of resp.Diagnostics.Append(xxx...).
funcs that return other values besides diag.Diagnostics have resp.Diagnostics.Append(diags) immediately following the function call.
funcs that return error have if err != nil { resp.Diagnostics.AddError("msg", err.Error()); return } right afterwards.

When will you add tests?

internal/providers/pluginfw/common/common.go Show resolved Hide resolved

common/client.go Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Show resolved Hide resolved

internal/providers/pluginfw/resources/volume/data_volumes.go Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/volume/data_volumes.go Outdated Show resolved Hide resolved


          comments

89419e5

mgyucht reviewed

View reviewed changes

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Outdated Show resolved Hide resolved

tanmay-db added 7 commits

August 29, 2024 16:15


          Update

959e184


          Update

0f87059


          Update

d44754b


          Merge branch 'main' of github.com:databricks/terraform-provider-datab…

c1a9776

…ricks into plugin-fw-resources-datasources

71d6f59

161a6c3

e976b44

tanmay-db requested a review from mgyucht

August 30, 2024 12:40


          Merge branch 'main' of github.com:databricks/terraform-provider-datab…

…ricks into plugin-fw-resources-datasources

tanmay-db commented

View reviewed changes

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go

+              	endpoint, err := w.QualityMonitors.GetByTableName(ctx, getMonitor.TableName.ValueString())
+              	if err != nil {
+              		if apierr.IsMissing(err) {
+              			resp.State.RemoveResource(ctx)

Contributor Author

tanmay-db Aug 30, 2024

highlight

mgyucht approved these changes

View reviewed changes

Contributor

mgyucht left a comment

Thank you for updating docs! Few bugs/nits, otherwise it is looking great.

CONTRIBUTING.md Outdated Show resolved Hide resolved

CONTRIBUTING.md Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/qualitymonitor/resource_quality_monitor.go Show resolved Hide resolved

internal/providers/pluginfw/resources/volume/data_volumes.go Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/volume/data_volumes.go Show resolved Hide resolved

tanmay-db and others added 6 commits

August 30, 2024 15:03


          Update CONTRIBUTING.md

065e6c4

Co-authored-by: Miles Yucht <miles@databricks.com>


          Update CONTRIBUTING.md

bfc62ce

Co-authored-by: Miles Yucht <miles@databricks.com>


          Update internal/providers/pluginfw/resources/qualitymonitor/resource_…

83553ed

…quality_monitor.go

Co-authored-by: Miles Yucht <miles@databricks.com>


          Update internal/providers/pluginfw/resources/volume/data_volumes.go

da03e87

Co-authored-by: Miles Yucht <miles@databricks.com>


          Update internal/providers/pluginfw/resources/volume/data_volumes.go

92cb9f5

Co-authored-by: Miles Yucht <miles@databricks.com>

5073d12

tanmay-db enabled auto-merge

August 30, 2024 13:08

tanmay-db added this pull request to the merge queue

Merged via the queue into main with commit f4687d6

7 checks passed

tanmay-db deleted the plugin-fw-resources-datasources branch

August 30, 2024 13:14

edwardfeng-db mentioned this pull request

[Internal] Add integration tests for volumes and quality monitor plugin framework #3975

Merged

5 tasks

github-merge-queue bot pushed a commit that referenced this pull request


          [Internal] Add integration tests for volumes and quality monitor plug…

f316f41

…in framework (#3975)

## Changes
<!-- Summary of your changes that are easy to understand -->
- Add integration tests for resource and data source introduced
[here](#3958)
- Small updates to `ConfigureDataSource` function to handle nil case
- In `data_volumes`, we don't support `computed` tf tag yet, so we need
to do that in customizable schema

## Tests
<!-- 
How is this tested? Please see the checklist below and also describe any
other relevant tests
-->

- [x] `make test` run locally
- [x] relevant change in `docs/` folder
- [x] covered with integration tests in `internal/acceptance`
- [x] relevant acceptance tests are passing
- [x] using Go SDK

hectorcast-db mentioned this pull request

[Release] Release v1.52.0 #4013

Closed

hectorcast-db added a commit that referenced this pull request


          [Release] Release v1.52.0

5206f57

### New Features and Improvements

 * Added `no_wait` option for clusters to skip waiting to start on cluster creation ([#3953](#3953)).
 * Introduced Plugin Framework ([#3920](#3920)).

### Bug Fixes

 * Add suppress diff for `azure_attributes.spot_bid_max_price` in `databricks_instance_pool` ([#3970](#3970)).
 * Correctly send workload_type fields in `databricks_cluster` to allow users to disable usage in certain contexts ([#3972](#3972)).
 * Fix `TestAccClusterResource_WorkloadType` ([#3989](#3989)).
 * Fix `databricks_sql_table` treatment of properties ([#3925](#3925)).
 * Force send fields for settings resources ([#3978](#3978)).
 * Handle cluster deletion in `databricks_library` read ([#3909](#3909)).
 * Make subscriptions optional for SqlAlertTask ([#3983](#3983)).

### Documentation

 * Add troubleshooting guide for Provider "registry.terraform.io/databricks/databricks" planned an invalid value ([#3961](#3961)).
 * Adopt official naming of Mosaic AI Vector Search ([#3971](#3971)).
 * Document Terraform 1.0 as minimum version ([#3952](#3952)).
 * Mention Salesforce as supported type in `databricks_connection` ([#3949](#3949)).
 * Reimplement Azure Databricks deployment guide to use VNet injection & NPIP ([#3986](#3986)).
 * Resolves [#3127](#3127): Remove deprecated account_id field from mws_credentials resource ([#3974](#3974)).
 * Small Grammar Corrections in Docs ([#4006](#4006)).
 * Update `databricks_vector_search_index` docs to match latest SDK ([#4008](#4008)).
 * Update aws_unity_catalog_assume_role_policy.md ([#3968](#3968)).
 * Update documentation regarding authentication with Azure-managed Service Principal using GITHUB OIDC ([#3932](#3932)).
 * Update metastore_assignment.md to properly reflect possible usage ([#3967](#3967)).
 * Update minimum supported terraform version to 1.1.5 ([#3965](#3965)).
 * Update resources diagram to include newer resources ([#3962](#3962)).
 * Update workspace_binding import command ([#3944](#3944)).
 * fix possible values for `securable_type` in `databricks_workspace_binding` ([#3942](#3942)).

### Internal Changes

 * Add `AddPlanModifer` method for AttributeBuilder ([#4009](#4009)).
 * Add integration tests for volumes and quality monitor plugin framework ([#3975](#3975)).
 * Add support for `computed` tag in TfSDK Structs ([#4005](#4005)).
 * Added `databricks_quality_monitor` resource and `databricks_volumes` data source to plugin framework ([#3958](#3958)).
 * Allow vector search tests to fail ([#3959](#3959)).
 * Make test utils public and move integration test for quality monitor ([#3993](#3993)).
 * Migrate Share resource to Go SDK ([#3916](#3916)).
 * Migrate imports for terraform plugin framework + update init test provider factory ([#3943](#3943)).
 * Move volumes test next to plugin framework data source ([#3995](#3995)).
 * Refactor provider and related packages ([#3940](#3940)).
 * Support import in acceptance test + adding import state for quality monitor ([#3994](#3994)).

### Dependency Updates

 * Bump github.com/hashicorp/hcl/v2 from 2.21.0 to 2.22.0 ([#3948](#3948)).
 * Update Go SDK to 0.46.0 ([#4007](#4007)).

### Exporter

 * Don't generate instance pools if the pool name is empty ([#3960](#3960)).

tanmay-db mentioned this pull request

[Release] Release v1.52.0 #4022

Merged

tanmay-db added a commit that referenced this pull request


          [Release] Release v1.52.0

c83c514

### New Features and Improvements

 * Add support for filters in `databricks_clusters` data source ([#4014](#4014)).
 * Added `no_wait` option for clusters to skip waiting to start on cluster creation ([#3953](#3953)).
 * Introduced Plugin Framework ([#3920](#3920)).
 * Library plugin framework migration ([#3979](#3979)).

### Bug Fixes

 * Add suppress diff for `azure_attributes.spot_bid_max_price` in `databricks_instance_pool` ([#3970](#3970)).
 * Correctly send workload_type fields in `databricks_cluster` to allow users to disable usage in certain contexts ([#3972](#3972)).
 * Fix `TestAccClusterResource_WorkloadType` ([#3989](#3989)).
 * Fix `databricks_sql_table` treatment of properties ([#3925](#3925)).
 * Force send fields for settings resources ([#3978](#3978)).
 * Handle cluster deletion in `databricks_library` read ([#3909](#3909)).
 * Make subscriptions optional for SqlAlertTask ([#3983](#3983)).
 * Permanently delete `ERROR` and `TERMINATED` state clusters if their creation fails ([#4021](#4021)).

### Documentation

 * Add troubleshooting guide for Provider "registry.terraform.io/databricks/databricks" planned an invalid value ([#3961](#3961)).
 * Adopt official naming of Mosaic AI Vector Search ([#3971](#3971)).
 * Document Terraform 1.0 as minimum version ([#3952](#3952)).
 * Mention Salesforce as supported type in `databricks_connection` ([#3949](#3949)).
 * Reimplement Azure Databricks deployment guide to use VNet injection & NPIP ([#3986](#3986)).
 * Resolves [#3127](#3127): Remove deprecated account_id field from mws_credentials resource ([#3974](#3974)).
 * Small Grammar Corrections in Docs ([#4006](#4006)).
 * Update `databricks_vector_search_index` docs to match latest SDK ([#4008](#4008)).
 * Update aws_unity_catalog_assume_role_policy.md ([#3968](#3968)).
 * Update documentation regarding authentication with Azure-managed Service Principal using GITHUB OIDC ([#3932](#3932)).
 * Update metastore_assignment.md to properly reflect possible usage ([#3967](#3967)).
 * Update minimum supported terraform version to 1.1.5 ([#3965](#3965)).
 * Update resources diagram to include newer resources ([#3962](#3962)).
 * Update workspace_binding import command ([#3944](#3944)).
 * fix possible values for `securable_type` in `databricks_workspace_binding` ([#3942](#3942)).

### Internal Changes

 * Add `AddPlanModifer` method for AttributeBuilder ([#4009](#4009)).
 * Add integration tests for volumes and quality monitor plugin framework ([#3975](#3975)).
 * Add support for `computed` tag in TfSDK Structs ([#4005](#4005)).
 * Added `databricks_quality_monitor` resource and `databricks_volumes` data source to plugin framework ([#3958](#3958)).
 * Allow vector search tests to fail ([#3959](#3959)).
 * Clean up comments in library resource ([#4015](#4015)).
 * Fix irregularities in plugin framework converter function errors ([#4010](#4010)).
 * Make test utils public and move integration test for quality monitor ([#3993](#3993)).
 * Migrate Share resource to Go SDK ([#3916](#3916)).
 * Migrate `databricks_cluster` data source to plugin framework ([#3988](#3988)).
 * Migrate imports for terraform plugin framework + update init test provider factory ([#3943](#3943)).
 * Move volumes test next to plugin framework data source ([#3995](#3995)).
 * Refactor provider and related packages ([#3940](#3940)).
 * Support import in acceptance test + adding import state for quality monitor ([#3994](#3994)).

### Dependency Updates

 * Bump github.com/hashicorp/hcl/v2 from 2.21.0 to 2.22.0 ([#3948](#3948)).
 * Update Go SDK to 0.46.0 ([#4007](#4007)).

### Exporter

 * Don't generate instance pools if the pool name is empty ([#3960](#3960)).

github-merge-queue bot pushed a commit that referenced this pull request


          [Release] Release v1.52.0 (#4022)

e13ce2c

### New Features and Improvements

* Add support for filters in `databricks_clusters` data source
([#4014](#4014)).
* Added `no_wait` option for clusters to skip waiting to start on
cluster creation
([#3953](#3953)).
* Introduced Plugin Framework
([#3920](#3920)).


### Bug Fixes

* Add suppress diff for `azure_attributes.spot_bid_max_price` in
`databricks_instance_pool`
([#3970](#3970)).
* Correctly send workload_type fields in `databricks_cluster` to allow
users to disable usage in certain contexts
([#3972](#3972)).
* Fix `databricks_sql_table` treatment of properties
([#3925](#3925)).
* Force send fields for settings resources
([#3978](#3978)).
* Handle cluster deletion in `databricks_library` read
([#3909](#3909)).
* Make subscriptions optional for SqlAlertTask
([#3983](#3983)).
* Permanently delete `ERROR` and `TERMINATED` state clusters if their
creation fails
([#4021](#4021)).


### Documentation

* Add troubleshooting guide for Provider
"registry.terraform.io/databricks/databricks" planned an invalid value
([#3961](#3961)).
* Adopt official naming of Mosaic AI Vector Search
([#3971](#3971)).
* Document Terraform 1.0 as minimum version
([#3952](#3952)).
* Mention Salesforce as supported type in `databricks_connection`
([#3949](#3949)).
* Reimplement Azure Databricks deployment guide to use VNet injection &
NPIP
([#3986](#3986)).
* Resolves
[#3127](#3127):
Remove deprecated account_id field from mws_credentials resource
([#3974](#3974)).
* Small Grammar Corrections in Docs
([#4006](#4006)).
* Update `databricks_vector_search_index` docs to match latest SDK
([#4008](#4008)).
* Update aws_unity_catalog_assume_role_policy.md
([#3968](#3968)).
* Update documentation regarding authentication with Azure-managed
Service Principal using GITHUB OIDC
([#3932](#3932)).
* Update metastore_assignment.md to properly reflect possible usage
([#3967](#3967)).
* Update minimum supported terraform version to 1.1.5
([#3965](#3965)).
* Update resources diagram to include newer resources
([#3962](#3962)).
* Update workspace_binding import command
([#3944](#3944)).
* fix possible values for `securable_type` in
`databricks_workspace_binding`
([#3942](#3942)).


### Internal Changes

* Add `AddPlanModifer` method for AttributeBuilder
([#4009](#4009)).
* Add integration tests for volumes and quality monitor plugin framework
([#3975](#3975)).
* Add support for `computed` tag in TfSDK Structs
([#4005](#4005)).
* Added `databricks_quality_monitor` resource and `databricks_volumes`
data source to plugin framework
([#3958](#3958)).
* Allow vector search tests to fail
([#3959](#3959)).
* Clean up comments in library resource
([#4015](#4015)).
* Fix irregularities in plugin framework converter function errors
([#4010](#4010)).
* Make test utils public and move integration test for quality monitor
([#3993](#3993)).
* Migrate Share resource to Go SDK
([#3916](#3916)).
* Migrate `databricks_cluster` data source to plugin framework
([#3988](#3988)).
* Migrate imports for terraform plugin framework + update init test
provider factory
([#3943](#3943)).
* Move volumes test next to plugin framework data source
([#3995](#3995)).
* Refactor provider and related packages
([#3940](#3940)).
* Support import in acceptance test + adding import state for quality
monitor
([#3994](#3994)).
* Library plugin framework migration
([#3979](#3979)).
* Fix `TestAccClusterResource_WorkloadType`
([#3989](#3989)).


### Dependency Updates

* Bump github.com/hashicorp/hcl/v2 from 2.21.0 to 2.22.0
([#3948](#3948)).
* Update Go SDK to 0.46.0
([#4007](#4007)).


### Exporter

* Don't generate instance pools if the pool name is empty
([#3960](#3960)).
* Expand list of non-interactive clusters
([#4023](#4023)).
* Ignore databricks_artifact_allowlist with zero artifact_matcher blocks
([#4019](#4019)).


## [Release] Release v1.51.0

### Breaking Changes

With this release, only protocol version 6 will be supported which is
compatible with terraform CLI version 1.1.5 and later. If you are using
an older version of the terraform CLI, please upgrade it to use this and
further releases of Databricks terraform provider.

### New Features and Improvements

* Automatically create `parent_path` folder when creating
`databricks_dashboard resource` if it doesn't exist
([#3778](#3778)).


### Bug Fixes

* Fixed logging for underlying Go SDK
([#3917](#3917)).
* Remove not necessary field in `databricks_job` schema
([#3907](#3907)).


### Internal Changes

* Add AttributeBuilder for Plugin Framework schema
([#3922](#3922)).
* Add CustomizableSchema for Plugin Framework
([#3927](#3927)).
* Add StructToSchema for Plugin Framework
([#3928](#3928)).
* Add codegen template and generated files for tfsdk structs
([#3911](#3911)).
* Add converter functions and tests for plugin framework
([#3914](#3914)).
* Added support to use protocol version 6 provider server for SDK plugin
([#3862](#3862)).
* Bump Go SDK to v0.45.0
([#3933](#3933)).
* Change name with the aliases in codegen template
([#3936](#3936)).
* Update jd version from latest to 1.8.1
([#3915](#3915)).
* Upgrade `staticcheck` to v0.5.1 to get Go 1.23 support
([#3931](#3931)).
* OPENAPI_SHA check
([#3935](#3935)).
* Use generic error for missing clusters
([#3938](#3938))


### Exporter

* Better support for notebooks with /Workspace path
([#3901](#3901)).
* Improve exporting of DLT and test coverage
([#3898](#3898)).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet