Improved registered_model handling for DAB deploy/destroy

[amichaelsen]

Describe the issue

We would like to include our Unity Catalog registered models in the Databricks Asset Bundles which is supported by the resources:registered_models:model syntax, however we run into issues when we want to modify/remove models from the bundle. In particular, if we first deploy a bundle with a registered model, run code that registers a version of that model, and then change the model name or which schema it is saved in, then when we redeploy the bundle with the new registered/model configuration it fails because it tries to delete the old registered model first, and cannot do so since it has registered versions. I don't expect the new registered model to preserve the versions of the old one, but I would like to be able to deploy the new bundle. The same issue would occur if I have a second model deployed with the bundle with registered versions that I later decide to drop from the project. I would need to delete the old model in the Unity Catalog before I could deploy the new bundle.

This is mentioned in this Databricks community thread.

Suggested fixes:

• delete registered models that are no longer in the bundle configuration (due to removal or name change)
• unbind all registered models that are no longer in the bundle configuration
• provide a flag for the CLI to force delete/unbind removed registered models

Configuration

bundle.yaml

grants: &grants
  grants:
    - privileges:
        - EXECUTE
      principal: ${workspace.current_user.userName}

variables:
  uc_registered_models:
    type: complex
    default:
      main_model: ${resources.registered_models.main_model.catalog_name}.${resources.registered_models.main_model.schema_name}.${resources.registered_models.main_model.name}

resources:
  registered_models:
    main_model:
      name: bundle_demo_model # must be all lower-case
      catalog_name: [catalog]
      schema_name: [schema]
      comment: Registered model in Unity Catalog for the DEMO repo.
      <<: *grants
  jobs:
    model_train:
      name: Train_Model
      tasks:
        - task_key: training
          notebook_task:
            notebook_path: notebooks/train_model
            base_parameters:
              model_name: ${var.uc_registered_models.main_model}
              experiment_path: /Users/${workspace.current_user.userName}/Bundled_Models_Demo_Experiment
            source: GIT
      git_source:
        git_url: ${bundle.git.origin_url}
        git_provider: gitHubEnterprise
        git_branch: ${bundle.git.branch}

targets:
  demo:
    default: true
    workspace:
      host: [databricks host url]

Model registration notebook (notebooks/train_model.py)

# Databricks notebook source

# MAGIC %pip install mlflow-skinny[databricks]

# COMMAND ----------

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
import mlflow
import json


# COMMAND ----------

dbutils.widgets.text("model_name", defaultValue="", label="UC Model Name")


params = dbutils.widgets.getAll()
print(json.dumps(params, indent=3))


# COMMAND ----------

mlflow.set_experiment(dbutils.widgets.get("experiment_path"))
with mlflow.start_run():
    # Train a sklearn model on the iris dataset
    X, y = datasets.load_iris(return_X_y=True, as_frame=True)
    clf = RandomForestClassifier(max_depth=7)
    clf.fit(X, y)
    # Take the first row of the training dataset as the model input example.
    input_example = X.iloc[[0]]
    # Log the model and register it as a new version in UC.
    mlflow.sklearn.log_model(
        sk_model=clf,
        artifact_path="model",
        # The signature is automatically inferred from the input example and its predicted output.
        input_example=input_example,
        registered_model_name=dbutils.widgets.get("model_name"),
    )

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy
2. Run databricks bundle run model_train
3. Change name: bundle_demo_model to name: bundle_demo_model_new
4. Run databricks bundle deploy
5. See error: Error: terraform apply: exit status 1 Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s)

Expected Behavior

Since we changed the model name, we would like the bundle to deploy with a registered model named bundle_demo_model_new. The old model bundle_demo_model can either be disconnected from the bundle or deleted.

Actual Behavior

The bundle deployment attempts to destroy the old registered model but since it has a version it throws an error and fails to deploy the bundle.

OS and CLI version

Databricks CLI v0.228.1
OS Sonoma 14.7 (23H124)

Is this a regression?

Not tested on previous versions.

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

15:18:56  INFO start pid=85406 version=0.228.1 args="databricks, bundle, deploy, --log-level=debug"
15:18:56 DEBUG Found bundle root at /Users/[user]/GitHub/databrics_bundle_demo (file /Users/[user]/GitHub/databrics_bundle_demo/bundle.yaml) pid=85406
15:18:56 DEBUG Apply pid=85406 mutator=load
15:18:56  INFO Phase: load pid=85406 mutator=load
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=EntryPoint
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=scripts.preinit
15:18:56 DEBUG No script defined for preinit, skipping pid=85406 mutator=load mutator=seq mutator=scripts.preinit
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=ProcessRootIncludes
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=VerifyCliVersion
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=EnvironmentsToTargets
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=InitializeVariables
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=LoadGitDetails
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=PythonMutator(load)
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=validate:unique_resource_keys
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=SelectDefaultTarget
15:18:56 DEBUG Apply pid=85406 mutator=load mutator=seq mutator=SelectDefaultTarget mutator=SelectTarget(demo)
15:18:56 DEBUG Apply pid=85406 mutator=<func>
15:18:56 DEBUG Apply pid=85406 mutator=<func>
15:18:56 DEBUG Apply pid=85406 mutator=seq
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize
15:18:56  INFO Phase: initialize pid=85406 mutator=seq mutator=initialize
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=RewriteSyncPaths
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=SyncDefaultPath
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=SyncInferRoot
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=MergeJobClusters
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=MergeJobParameters
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=MergeJobTasks
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=MergePipelineClusters
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
15:18:56 DEBUG Loading profile DEFAULT because of host match pid=85406
15:18:56 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
15:18:56 DEBUG Loading profile DEFAULT because of host match pid=85406
15:18:57 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "displayName": "***************",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "[user]"
<     }
<   ],
<   "externalId": "***************",
<   "groups": [
<     {
<       "$ref": "Groups/***************",
<       "display": "[schema]-***************",
<       "type": "direct",
<       "value": "***************"
<     }
<   ],
<   "id": "***************",
<   "name": {
<     "familyName": "***************",
<     "givenName": "***************"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
<   ],
<   "userName": "[user]"
< } pid=85406 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=SetVariables
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ResolveResourceReferences
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=SetRunAs
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=OverrideCompute
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ApplyPresets
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=DefaultQueueing
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ExpandPipelineGlobPaths
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ConfigureWSFS
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=TranslatePaths
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=ApplyBundlePermissions
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=FilterCurrentUserFromPermissions
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotateJobs
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotatePipelines
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
15:18:57 DEBUG DATABRICKS_TF_EXEC_PATH is not defined pid=85406 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
15:18:57 DEBUG Using Terraform at /Users/[user]/GitHub/databrics_bundle_demo/.databricks/bundle/demo/bin/terraform pid=85406 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
15:18:57 DEBUG DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=85406 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
15:18:57 DEBUG Environment variables for Terraform: PATH, TMPDIR, DATABRICKS_USER_AGENT_EXTRA, DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_AUTH_TYPE, HOME pid=85406 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
15:18:57 DEBUG No script defined for postinit, skipping pid=85406 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build
15:18:57  INFO Phase: build pid=85406 mutator=seq mutator=build
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
15:18:57 DEBUG No script defined for prebuild, skipping pid=85406 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
15:18:57  INFO No local tasks in databricks.yml config, skipping auto detect pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=scripts.postbuild
15:18:57 DEBUG No script defined for postbuild, skipping pid=85406 mutator=seq mutator=build mutator=seq mutator=scripts.postbuild
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=build mutator=seq mutator=ResolveVariableReferences
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=deploy
15:18:57  INFO Phase: deploy pid=85406 mutator=seq mutator=deploy
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.predeploy
15:18:57 DEBUG No script defined for predeploy, skipping pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.predeploy
15:18:57 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire
15:18:57  INFO Acquiring deployment lock (force: false) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire
15:18:57 DEBUG POST /api/2.0/workspace-files/import-file/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock?overwrite=false
> {
>   "AcquisitionTime": "2024-10-21T15:18:57.040287-07:00",
>   "ID": "***************",
>   "IsForced": false,
>   "User": "[user]"
> }
< HTTP/2.0 200 OK pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire sdk=true
15:18:58 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1729549137624,
<   "modified_at": 1729549137624,
<   "object_id": ***************,
<   "object_type": "FILE",
<   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock",
<   "resource_id": "***************"
< } pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire sdk=true
15:18:58 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock
< HTTP/2.0 200 OK
< <Streaming response> pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire sdk=true
15:18:58 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
15:18:58 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq
15:18:58 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
15:18:58 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/terraform.tfstate&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1729544782753,
<   "modified_at": 1729546946972,
<   "object_id": ***************,
<   "object_type": "FILE",
<   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/state/terraform.tfstate",
<   "resource_id": "***************"
< } pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
15:18:58 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/terraform.tfstate
< HTTP/2.0 200 OK
< <Streaming response> pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
15:18:58 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
15:18:58  INFO Opening remote deployment state file pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
15:18:59 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deployment.json&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1729544773119,
<   "modified_at": 1729546942599,
<   "object_id": ***************,
<   "object_type": "FILE",
<   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deployment.json",
<   "resource_id": "***************"
< } pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull sdk=true
15:18:59 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deployment.json
< HTTP/2.0 200 OK
< <Streaming response> pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull sdk=true
15:18:59  INFO Local deployment state is the same or newer, ignoring remote state pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=ValidateGitDetails
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp
15:18:59 ERROR failed to delete /Users/[user]/.bundle/bundle_w_models_demo/demo/artifacts/.internal: unable to delete filer root pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp
15:18:59 DEBUG POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/artifacts/.internal"
> }
< HTTP/2.0 200 OK
< {} pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp sdk=true
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=libraries.ExpandGlobReferences
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=libraries.Upload
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=NoOp
15:18:59 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload
Uploading bundle files to /Users/[user]/.bundle/bundle_w_models_demo/demo/files...
15:18:59 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[user]/.bundle/bundle_w_models_demo/demo/files
< HTTP/2.0 200 OK
< {
<   "object_id": ***************,
<   "object_type": "DIRECTORY",
<   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/files",
<   "resource_id": "***************"
< } pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload sdk=true
15:18:59 DEBUG Path /Users/[user]/.bundle/bundle_w_models_demo/demo/files has type directory (ID: 3438986019336085) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload
15:19:00 DEBUG POST /api/2.0/workspace-files/import-file/Users/[user]/.bundle/bundle_w_models_demo/demo/files/notebooks/train_model.py?overwrite=true
> # Databricks notebook source
> 
> # MAGIC %pip install mlflow-skinny[databricks]
> 
> # COMMAND --------... (1109 more bytes)
< HTTP/2.0 200 OK pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload sdk=true
15:19:00  INFO Uploaded bundle files pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload
15:19:00 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-update
15:19:00  INFO Loading deployment state from /Users/[user]/GitHub/databrics_bundle_demo/.databricks/bundle/demo/deployment.json pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-update
15:19:00 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-push
15:19:00  INFO Writing local deployment state file to remote state directory pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-push
15:19:01 DEBUG POST /api/2.0/workspace-files/import-file/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deployment.json?overwrite=true
> {
>   "cli_version": "0.228.1",
>   "files": [
>     {
>       "is_notebook": true,
>       "local_path": "notebooks/train_model.py"
>     },
>     {
>       "is_notebook": false,
>       "local_path": ".gitignore"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "README.md"
>     },
>     {
>       "is_notebook": false,
>       "local_path": "bundle.yaml"
>     }
>   ],
>   "id": "***************",
>   "seq": 11,
>   "timestamp": "2024-10-21T22:19:00.669053Z",
>   "version": 1
> }
< HTTP/2.0 200 OK pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-push sdk=true
15:19:01 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=ApplyWorkspaceRootPermissions
15:19:01 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Interpolate
15:19:01 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Write
15:19:01 DEBUG registered model normalization diagnostic: unknown field: grants pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Write
15:19:01 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=check-running-resources
15:19:01 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Plan
15:19:02 DEBUG Planning complete and persisted at /Users/[user]/GitHub/databrics_bundle_demo/.databricks/bundle/demo/terraform/plan
 pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Plan
15:19:02 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If
15:19:03 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred
15:19:03 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq
15:19:03 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=log_string
Deploying resources...
15:19:03 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=terraform.Apply
15:19:03 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=terraform.Apply
15:19:03 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq
15:19:03 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq
15:19:03 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=terraform:state-push
Updating deployment state...
15:19:03  INFO Writing local state file to remote state directory pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=terraform:state-push
15:19:04 DEBUG POST /api/2.0/workspace-files/import-file/Users/[user]/.bundle/bundle_w_models_demo/demo/state/terraform.tfstate?overwrite=true
> {
>   "check_results": null,
>   "lineage": "***************",
>   "outputs": {},
>   "resources": [
>     {
>       "instances": null,
>       "mode": "managed",
>       "name": "registered_model_main_model",
>       "provider": "provider[\"registry.terraform.io/databricks/databricks\"]",
>       "type": "databricks_grants"
>     },
>     {
>       "instances": null,
>       "mode": "managed",
>       "name": "model_train",
>       "provider": "provider[\"registry.terraform.io/databricks/databricks\"]",
>       "type": "databricks_job"
>     },
>     {
>       "instances": [
>         {
>           "attributes": {
>             "catalog_name": "[catalog]",
>             "comment": "Registered model in Unity Catalog for the DEMO repo.",
>             "id": "[catalog].[schema].bundle_demo_model",
>             "name": "bundle_demo_model",
>             "owner": "[user]",
>             "schema_name": "[schema]",
>             "storage_location": "***************.dfs.core.windows.net/[schema]/models/***************... (5 more bytes)"
>           },
>           "private": "***************",
>           "schema_version": 0,
>           "sensitive_attributes": null
>         }
>       ],
>       "mode": "managed",
>       "name": "main_model",
>       "provider": "provider[\"registry.terraform.io/databricks/databricks\"]",
>       "type": "databricks_registered_model"
>     }
>   ],
>   "serial": 29,
>   "terraform_version": "1.5.5",
>   "version": 4
> }
< HTTP/2.0 200 OK pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=terraform:state-push sdk=true
15:19:04 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=terraform.Load
15:19:04 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=metadata.Compute
15:19:04 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=metadata.Upload
15:19:05 DEBUG POST /api/2.0/workspace-files/import-file/Users/[user]/.bundle/bundle_w_models_demo/demo/state/metadata.json?overwrite=true
> {
>   "config": {
>     "bundle": {
>       "git": {
>         "branch": "demo/models-in-bundle-configuration",
>         "bundle_root_path": ".",
>         "commit": "ee4db0b6f9613f9511a9ba0b6d353ec5788abeb3",
>         "origin_url": "[github]/databricks_bundle_demo.git"
>       }
>     },
>     "resources": {
>       "jobs": {
>         "model_train": {
>           "relative_path": "bundle.yaml"
>         }
>       }
>     },
>     "workspace": {
>       "file_path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/files"
>     }
>   },
>   "version": 1
> }
< HTTP/2.0 200 OK pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=metadata.Upload sdk=true
15:19:05 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred mutator=seq mutator=log_string
Deployment complete!
15:19:05 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If mutator=deferred
15:19:05 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=If
15:19:05 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq
15:19:05 DEBUG Apply pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
15:19:05  INFO Releasing deployment lock pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
15:19:05 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1729549137624,
<   "modified_at": 1729549137624,
<   "object_id": ***************,
<   "object_type": "FILE",
<   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock",
<   "resource_id": "***************"
< } pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
15:19:06 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1729549137624,
<   "modified_at": 1729549137624,
<   "object_id": ***************,
<   "object_type": "FILE",
<   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock",
<   "resource_id": "***************"
< } pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
15:19:06 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock
< HTTP/2.0 200 OK
< <Streaming response> pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
15:19:06 DEBUG POST /api/2.0/workspace/delete
> {
>   "path": "/Users/[user]/.bundle/bundle_w_models_demo/demo/state/deploy.lock"
> }
< HTTP/2.0 200 OK
< {} pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
15:19:06 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
15:19:06 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq mutator=seq
15:19:06 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy mutator=seq
15:19:06 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq mutator=deploy
15:19:06 ERROR Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s) pid=85406 mutator=seq
Error: terraform apply: exit status 1Error: cannot delete registered model: Function '[catalog].[schema].bundle_demo_model' is not empty. The function has 1 model versions(s)15:19:06 ERROR failed execution pid=85406 exit_code=1 error=AlreadyPrinted