removing modules, updating explicit depends on, updating SRA, and more

aws/README.md

@@ -21,7 +21,8 @@ There are four separate operation modes you can choose for the underlying networ
 
 - **Sandbox**: Sandbox or open egress. Selecting 'sandbox' as the operation mode allows traffic to flow freely to the public internet. This mode is suitable for sandbox or development scenarios where data exfiltration protection is of minimal concern, and developers need to access public APIs, packages, and more.
 
-- **Firewall**: Firewall or limited egress. Choosing 'firewall' as the operation mode permits traffic flow only to a selected list of public addresses. This mode is applicable in situations where open internet access is necessary for certain tasks, but unfiltered traffic is not an option due to the sensitivity of the workloads or data. **NOTE**: Due to a limitation in the AWS Network Firewall's ability to use fully qualified domain names for non-HTTP/HTTPS traffic, an external data source is required for the external Hive metastore. For sensitive production workloads, it is recommended to use isolated operation mode and Unity Catalog, a self-hosted Hive metastore, or to explore other firewall services to address AWS Network Firewall's limitations.
+- **Firewall**: Firewall or limited egress. Choosing 'firewall' as the operation mode permits traffic flow only to a selected list of public addresses. This mode is applicable in situations where open internet access is necessary for certain tasks, but unfiltered traffic is not an option due to the sensitivity of the workloads or data. 
+    - **WARNING**: Due to a limitation in AWS Network Firewall's support for fully qualified domain names (FQDNs) in non-HTTP/HTTPS traffic, an IP address is required to allow communication with the Hive Metastore. This dependency on a static IP introduces the potential for downtime if the Hive Metastore's IP changes. For sensitive production workloads, it is recommended to explore the isolated operation mode or consider alternative firewall solutions that provide better handling of dynamic IPs or FQDNs.
 
 - **Isolated**:  Isolated or no egress. Opting for 'isolated' as the operation mode prevents any traffic to the public internet. Traffic is limited to AWS private endpoints, either to AWS services or the Databricks control plane. This mode should be used in cases where access to the public internet is completely unsupported. **NOTE**: Apache Derby Metastore will be required for clusters and non-serverless SQL Warehouses. For more information, please view this [knowledge article](https://kb.databricks.com/metastore/set-up-embedded-metastore).
 
@@ -45,18 +46,10 @@ See the below networking diagrams for more information.
 - **Unity Catalog**: [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/index.html) is a unified governance solution for all data and AI assets including files, tables, and machine learning models. Unity Catalog provides a modern approach to granular access controls with centralized policy, auditing, and lineage tracking - all integrated into your Databricks workflow. **NOTE**: SRA creates a workspace specific catalog that is isolated to that individual workspace. To change these settings please update uc_catalog.tf under the workspace_security_modules.
 
 
-## Post Workspace Deployment
-
-- **Service Principals**: A [Service principal](https://docs.databricks.com/administration-guide/users-groups/service-principals.html) is an identity that you create in Databricks for use with automated tools, jobs, and applications. It's against best practice to tie production workloads to individual user accounts, and so we recommend configuring these service principals within Databricks. In this template, we create an example service principal.
-
-- **Token Management**: [Personal access tokens](https://docs.databricks.com/dev-tools/api/latest/authentication.html) are used to access Databricks REST APIs in-lieu of passwords. In this template we create an example token and set its time-to-live. This can be set at an administrative level for all users.
-
-- **Secret Management** Integrating with heterogeneous systems requires managing a potentially large set of credentials and safely distributing them across an organization. Instead of directly entering your credentials into a notebook, use [Databricks secrets](https://docs.databricks.com/security/secrets/index.html) to store your credentials and reference them in notebooks and jobs. In this template, we create an example secret.
-
-
 ## Optional Deployment Configurations
 
 - **Audit and Billable Usage Logs**: Databricks delivers logs to your S3 buckets. [Audit logs](https://docs.databricks.com/administration-guide/account-settings/audit-logs.html) contain two levels of events: workspace-level audit logs with workspace-level events and account-level audit logs with account-level events. In addition to these logs, you can generate additional events by enabling verbose audit logs. [Billable usage logs](https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html) are delivered daily to an AWS S3 storage bucket. There will be a separate CSV file for each workspace. This file contains historical data about the workspace's cluster usage in Databricks Units (DBUs).
+- **System Tables Schemas**: System Tables provide visiblity into access, billing, compute, Lakeflow, and storage logs. These tables can be found within the system catalog in Unity Catalog.
 
 - **Cluster Example**: An example of a cluster and a cluster policy has been included. **NOTE:** Please be aware this will create a cluster within your Databricks workspace including the underlying EC2 instance.
 
@@ -80,11 +73,6 @@ See the below networking diagrams for more information.
 - **Audit Log Alerting**: Audit Log Alerting, based on this [blog post](https://www.databricks.com/blog/improve-lakehouse-security-monitoring-using-system-tables-databricks-unity-catalog), creates 40+ SQL alerts to monitor for incidents based on a Zero Trust Architecture (ZTA) model. **NOTE:** Please be aware this creates a cluster, a job, and queries within your environment. 
 
 
-## Public Preview Features
-
-- **System Tables Schemas**: System Table schemas are currently in private preview. System Tables provide visiblity into access, billing, compute, and storage logs. In this deployment the metastore admin, service principle, owns the table. Additional grant statements will be needed. **NOTE:** Please note this is currently in public preview.
-
-
 ## Additional Security Recommendations and Opportunities
 
 In this section, we break down additional security recommendations and opportunities to maintain a strong security posture that either cannot be configured into this Terraform script or is very specific to individual customers (e.g. SCIM, SSO, Front-End PrivateLink, etc.)
@@ -109,11 +97,12 @@ In this section, we break down additional security recommendations and opportuni
 3. Decide which [operation](https://github.com/databricks/terraform-databricks-sra/tree/main/aws/tf#operation-mode) mode you'd like to use.
 4. Fill out `sra.tf` in place
 5. Fill out `template.tfvars.example` remove the .example part of the file name
-6. CD into `tf`
-7. Run `terraform init`
-8. Run `terraform validate`
-9. From `tf` directory, run `terraform plan -var-file ../example.tfvars`
-10. Run `terraform apply -var-file ../example.tfvars`
+6. Configure the [AWS](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration) and [Databricks](https://registry.terraform.io/providers/databricks/databricks/latest/docs#authentication) provider authentication
+7. CD into `tf`
+8. Run `terraform init`
+9. Run `terraform validate`
+10. From `tf` directory, run `terraform plan -var-file ../example.tfvars`
+11. Run `terraform apply -var-file ../example.tfvars`
 
 
 ## Network Diagram - Sandbox

aws/tf/modules/sra/databricks_account.tf

@@ -15,7 +15,6 @@ module "log_delivery" {
 
 // Create Unity Catalog Metastore - No Root Storage
 module "uc_init" {
-  count  = var.metastore_exists == false ? 1 : 0
   source = "./databricks_account/uc_init"
   providers = {
     databricks = databricks.mws
@@ -26,6 +25,7 @@ module "uc_init" {
   resource_prefix       = var.resource_prefix
   region                = var.region
   metastore_name        = join("", [var.resource_prefix, "-", var.region, "-", "uc"])
+  metastore_exists      = var.metastore_exists
 }
 
 // Unity Catalog Assignment
@@ -35,12 +35,10 @@ module "uc_assignment" {
     databricks = databricks.mws
   }
 
-  metastore_id = var.metastore_exists ? null : module.uc_init[0].metastore_id
+  metastore_id = module.uc_init.metastore_id
   region       = var.region
   workspace_id = module.databricks_mws_workspace.workspace_id
-  depends_on = [
-    module.databricks_mws_workspace
-  ]
+  depends_on   = [module.databricks_mws_workspace, module.uc_init]
 }
 
 // Create Databricks Workspace
@@ -66,22 +64,6 @@ module "databricks_mws_workspace" {
   workspace_storage_key_alias = aws_kms_alias.workspace_storage_key_alias.name
 }
 
-// Service Principal
-module "service_principal" {
-  source = "./databricks_account/service_principal"
-  providers = {
-    databricks = databricks.mws
-  }
-
-  created_workspace_id                   = module.databricks_mws_workspace.workspace_id
-  workspace_admin_service_principal_name = var.workspace_admin_service_principal_name
-
-  depends_on = [
-    module.databricks_mws_workspace,
-    module.uc_assignment
-  ]
-}
-
 // User Workspace Assignment (Admin)
 module "user_assignment" {
   source = "./databricks_account/user_assignment"
@@ -91,9 +73,5 @@ module "user_assignment" {
 
   created_workspace_id = module.databricks_mws_workspace.workspace_id
   workspace_access     = var.user_workspace_admin
-
-  depends_on = [
-    module.databricks_mws_workspace,
-    module.uc_assignment
-  ]
+  depends_on           = [module.uc_assignment, module.databricks_mws_workspace]
 }

aws/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf

@@ -1,11 +1,7 @@
 // Metastore Assignment
 
-data "databricks_metastore" "this" {
-  region = var.region
-}
-
 resource "databricks_metastore_assignment" "default_metastore" {
   workspace_id         = var.workspace_id
-  metastore_id         = var.metastore_id == null ? data.databricks_metastore.this.id : var.metastore_id
+  metastore_id         = var.metastore_id
   default_catalog_name = "hive_metastore"
 }

aws/tf/modules/sra/databricks_account/uc_init/outputs.tf

@@ -1,3 +1,3 @@
 output "metastore_id" {
-  value = databricks_metastore.this.id
+  value = var.metastore_exists ? data.databricks_metastore.this[0].id : databricks_metastore.this[0].id
 }

aws/tf/modules/sra/databricks_account/uc_init/uc_init.tf

@@ -1,7 +1,13 @@
 // Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog
 
-// Metastore
+// Optional data source - only run if the metastore exists
+data "databricks_metastore" "this" {
+  count  = var.metastore_exists ? 1 : 0
+  region = var.region
+}
+
 resource "databricks_metastore" "this" {
+  count         = var.metastore_exists ? 0 : 1
   name          = "${var.resource_prefix}-${var.region}-unity-catalog"
   region        = var.region
   force_destroy = true

aws/tf/modules/sra/databricks_account/uc_init/variables.tf

@@ -6,6 +6,10 @@ variable "databricks_account_id" {
   type = string
 }
 
+variable "metastore_exists" {
+  type = string
+}
+
 variable "metastore_name" {
   type = string
 }

aws/tf/modules/sra/databricks_workspace.tf

@@ -15,9 +15,7 @@ module "uc_catalog" {
   workspace_id                 = module.databricks_mws_workspace.workspace_id
   user_workspace_catalog_admin = var.user_workspace_catalog_admin
 
-  depends_on = [
-    module.databricks_mws_workspace, module.uc_assignment
-  ]
+  depends_on = [module.databricks_mws_workspace, module.uc_assignment]
 }
 
 // Create Read-Only Storage Location for Data Bucket & External Location
@@ -33,10 +31,6 @@ module "uc_external_location" {
   resource_prefix                   = var.resource_prefix
   read_only_data_bucket             = var.read_only_data_bucket
   read_only_external_location_admin = var.read_only_external_location_admin
-
-  depends_on = [
-    module.databricks_mws_workspace, module.uc_assignment
-  ]
 }
 
 // Workspace Admin Configuration
@@ -46,34 +40,6 @@ module "admin_configuration" {
   providers = {
     databricks = databricks.created_workspace
   }
-
-  depends_on = [
-    module.databricks_mws_workspace
-  ]
-}
-
-// Token Management
-module "token_management" {
-  source = "./databricks_workspace/workspace_security_modules/token_management"
-  providers = {
-    databricks = databricks.created_workspace
-  }
-
-  depends_on = [
-    module.databricks_mws_workspace
-  ]
-}
-
-// Secret Management
-module "secret_management" {
-  source = "./databricks_workspace/workspace_security_modules/secret_management"
-  providers = {
-    databricks = databricks.created_workspace
-  }
-
-  depends_on = [
-    module.databricks_mws_workspace
-  ]
 }
 
 // IP Access Lists - Optional
@@ -85,10 +51,6 @@ module "ip_access_list" {
   }
 
   ip_addresses = var.ip_addresses
-
-  depends_on = [
-    module.databricks_mws_workspace
-  ]
 }
 
 // Create Create Cluster - Optional
@@ -100,25 +62,17 @@ module "cluster_configuration" {
   }
 
   compliance_security_profile_egress_ports = var.compliance_security_profile_egress_ports
-  secret_config_reference                  = module.secret_management.config_reference
   resource_prefix                          = var.resource_prefix
   operation_mode                           = var.operation_mode
-  depends_on = [
-    module.databricks_mws_workspace, module.secret_management
-  ]
 }
 
-// Public Preview - System Table Schemas - Optional
-module "public_preview_system_table" {
-  source = "./databricks_workspace/public_preview/system_schema/"
+// System Table Schemas Enablement - Optional
+module "system_table" {
+  source = "./databricks_workspace/workspace_security_modules/system_schema/"
   count  = var.enable_system_tables_schema_boolean ? 1 : 0
   providers = {
     databricks = databricks.created_workspace
   }
-
-  depends_on = [
-    module.databricks_mws_workspace
-  ]
 }
 
 // SAT Implementation - Optional
@@ -129,16 +83,18 @@ module "security_analysis_tool" {
     databricks = databricks.created_workspace
   }
 
-  databricks_url     = module.databricks_mws_workspace.workspace_url
-  workspace_PAT      = module.service_principal.service_principal_id
-  workspace_id       = module.databricks_mws_workspace.workspace_id
-  account_console_id = var.databricks_account_id
-  client_id          = var.client_id
-  client_secret      = var.client_secret
-  use_sp_auth        = true
+  databricks_url       = module.databricks_mws_workspace.workspace_url
+  workspace_id         = module.databricks_mws_workspace.workspace_id
+  account_console_id   = var.databricks_account_id
+  client_id            = var.client_id
+  client_secret        = var.client_secret
+  use_sp_auth          = true
+  proxies              = {}
+  analysis_schema_name = "SAT"
+
 
   depends_on = [
-    module.databricks_mws_workspace, module.service_principal
+    module.databricks_mws_workspace
   ]
 }
 
@@ -151,8 +107,4 @@ module "audit_log_alerting" {
   }
 
   alert_emails = [var.user_workspace_admin]
-
-  depends_on = [
-    module.databricks_mws_workspace, module.uc_assignment
-  ]
 }

aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf

@@ -7,8 +7,10 @@ terraform {
 }
 
 module "common" {
-  source             = "../common/"
-  account_console_id = var.account_console_id
-  workspace_id       = var.workspace_id
-  sqlw_id            = var.sqlw_id
+  source               = "../common/"
+  account_console_id   = var.account_console_id
+  workspace_id         = var.workspace_id
+  sqlw_id              = var.sqlw_id
+  analysis_schema_name = var.analysis_schema_name
+  proxies              = var.proxies
 }

aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf

@@ -1,5 +1,17 @@
 ### AWS Specific Secrets
 
+resource "databricks_secret" "user" {
+  key          = "user"
+  string_value = var.account_user
+  scope        = module.common.secret_scope_id
+}
+
+resource "databricks_secret" "pass" {
+  key          = "pass"
+  string_value = var.account_pass
+  scope        = module.common.secret_scope_id
+}
+
 resource "databricks_secret" "use_sp_auth" {
   key          = "use-sp-auth"
   string_value = var.use_sp_auth