diff --git a/README.md b/README.md index f2beed5..967c638 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,14 @@ # Security Reference Architectures (SRA) - Terraform Templates +

+ +

## Project Overview Security Reference Architecture (SRA) with Terraform templates makes deploying workspaces with Security Best Practices easy. You can programmatically deploy workspaces and the required cloud infrastructure using the official Databricks Terraform provider. These unified Terraform templates are pre-configured with hardened security settings similar to our most security-conscious customers. The initial templates based on [Databricks Security Best Practices](https://www.databricks.com/trust/security-features#best-practices) - [AWS](https://github.com/databricks/terraform-databricks-sra/tree/main/aws) +- [AWS Govcloud](https://github.com/databricks/terraform-databricks-sra/tree/main/aws-gov) - [Azure](https://github.com/databricks/terraform-databricks-sra/tree/main/azure) - [GCP](https://github.com/databricks/terraform-databricks-sra/tree/main/gcp) diff --git a/aws-gov/.gitignore b/aws-gov/.gitignore new file mode 100644 index 0000000..1a768a0 --- /dev/null +++ b/aws-gov/.gitignore @@ -0,0 +1,43 @@ +# Local .terraform directories +*/.terraform/* +*/.terraform +.terraform.lock.hcl + +# .tfstate files +*.tfstate +*.tfstate.* + +# environment file +aws/example.tvars + +# Crash log files +crash.log + +# Ignore CLI configuration files +.terraformrc terraform.rc + +# Ignore any .tfvars files that are generated automatically for each Terraform run. Most +# .tfvars files are managed as part of configuration and so should be included in +# version control. +# +*auto.tfvars + +# Ignore override files as they are usually used to override resources locally and so +# are not checked in +override.tf +override.tf.json +*_override.tf +*_override.tf.json + +# Include override files you do wish to add to version control using negated pattern +# +# !example_override.tf + +# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan +# example: *tfplan* + +# MAC Control +.DS_Store + +# IntelliJ +.idea/ \ No newline at end of file diff --git a/aws-gov/README.md b/aws-gov/README.md new file mode 100644 index 0000000..ed34ca2 --- /dev/null +++ b/aws-gov/README.md @@ -0,0 +1,126 @@ +# Security Reference Architectures (SRA) - Terraform Templates + + +## Introduction + +Databricks has worked with thousands of customers to securely deploy the Databricks platform with appropriate security features to meet their architecture requirements. + +This Security Reference Architecture (SRA) repository implements common security features as a unified terraform templates that are typically deployed by our security conscious customers. + + +## Component Breakdown and Description + +In this section, we break down each of the components that we've included in this Security Reference Architecture. + +In various `.tf` scripts, we have included direct links to the Databricks Terraform documentation. The [official documentation](https://registry.terraform.io/providers/databricks/databricks/latest/docs) can be found here. + + +## Operation Mode: + +There are four separate operation modes you can choose for the underlying network configurations of your workspaces: **sandbox**, **firewall**, **isolated**, and **custom**. + +- **Sandbox**: Sandbox or open egress. Selecting 'sandbox' as the operation mode allows traffic to flow freely to the public internet. This mode is suitable for sandbox or development scenarios where data exfiltration protection is of minimal concern, and developers need to access public APIs, packages, and more. + +- **Firewall**: Firewall or limited egress. Choosing 'firewall' as the operation mode permits traffic flow only to a selected list of public addresses. This mode is applicable in situations where open internet access is necessary for certain tasks, but unfiltered traffic is not an option due to the sensitivity of the workloads or data. + - **WARNING**: Due to a limitation in AWS Network Firewall's support for fully qualified domain names (FQDNs) in non-HTTP/HTTPS traffic, an IP address is required to allow communication with the Hive Metastore. This dependency on a static IP introduces the potential for downtime if the Hive Metastore's IP changes. For sensitive production workloads, it is recommended to explore the isolated operation mode or consider alternative firewall solutions that provide better handling of dynamic IPs or FQDNs. + +- **Isolated**: Isolated or no egress. Opting for 'isolated' as the operation mode prevents any traffic to the public internet. Traffic is limited to AWS private endpoints, either to AWS services or the Databricks control plane. This mode should be used in cases where access to the public internet is completely unsupported. **NOTE**: Apache Derby Metastore will be required for clusters and non-serverless SQL Warehouses. For more information, please view this [knowledge article](https://kb.databricks.com/metastore/set-up-embedded-metastore). + +- **Custom**: Custom or bring your own network. Selecting 'custom' allows you to input your own details for a VPC ID, subnet IDs, security group IDs, and PrivateLink endpoint IDs. This mode is recommended when networking assets are created in different pipelines or are pre-assigned to a team by a centralized infrastructure team. + +See the below networking diagrams for more information. + + +## Infrastructure Deployment + +- **Customer-managed VPC**: A [customer-managed VPC](https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html) allows Databricks customers to exercise more control over network configuration to comply with specific cloud security and governance standards that a customer's organization may require. + +- **AWS VPC Endpoints for S3, STS, and Kinesis**: Using AWS PrivateLink technology, a VPC endpoint is a service that connects a customer's VPC endpoint to AWS services without traversing public IP addresses. [S3, STS, and Kinesis endpoints](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#step-5-add-vpc-endpoints-for-other-aws-services-recommended-but-optional) are best practices for standard enterprise Databricks deployments. Additional endpoints can be configured depending on use case (e.g. Amazon DynamoDB and AWS Glue). + +- **Back-end AWS PrivateLink Connectivity**: AWS PrivateLink provides a private network route from one AWS environment to another. [Back-end PrivateLink](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html#overview) is configured so that communication between the customer's data plane and the Databricks control plane does not traverse public IP addresses. This is accomplished through Databricks specific interface VPC endpoints. Front-end PrivateLink is available as well for customers to ensure users traffic remains over the AWS backbone. However front-end PrivateLink is not included in this Terraform template. + +- **Scoped-down IAM Policy for the Databricks cross-account role**: A [cross-account role](https://docs.databricks.com/administration-guide/account-api/iam-role.html) is needed for users, jobs, and other third-party tools to spin up Databricks clusters within the customer's data plane environment. This cross-account role can be scoped down to only function within the parameters of the data plane's VPC, subnets, and security group. + +- **Restrictive Root Bucket**: Each workspace, prior to creation, registers a [dedicated S3 bucket](https://docs.databricks.com/administration-guide/account-api/aws-storage.html). This bucket is for workspace assets. On AWS, S3 bucket policies can be applied to limit access to the Databricks control plane and the customer data plane. + +- **Unity Catalog**: [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/index.html) is a unified governance solution for all data and AI assets including files, tables, and machine learning models. Unity Catalog provides a modern approach to granular access controls with centralized policy, auditing, and lineage tracking - all integrated into your Databricks workflow. **NOTE**: SRA creates a workspace specific catalog that is isolated to that individual workspace. To change these settings please update uc_catalog.tf under the workspace_security_modules. + + +## Optional Deployment Configurations + +- **Audit and Billable Usage Logs**: Databricks delivers logs to your S3 buckets. [Audit logs](https://docs.databricks.com/administration-guide/account-settings/audit-logs.html) contain two levels of events: workspace-level audit logs with workspace-level events and account-level audit logs with account-level events. In addition to these logs, you can generate additional events by enabling verbose audit logs. [Billable usage logs](https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html) are delivered daily to an AWS S3 storage bucket. There will be a separate CSV file for each workspace. This file contains historical data about the workspace's cluster usage in Databricks Units (DBUs). +- **System Tables Schemas**: System Tables provide visiblity into access, billing, compute, Lakeflow, and storage logs. These tables can be found within the system catalog in Unity Catalog. + +- **Cluster Example**: An example of a cluster and a cluster policy has been included. **NOTE:** Please be aware this will create a cluster within your Databricks workspace including the underlying EC2 instance. + +- **IP Access Lists**: IP Access can be enabled to only allow a subset of IPs to access the Databricks workspace console. **NOTE:** Please verify all of the IPs are correct prior to enabling this feature to prevent a lockout scenario. + +- **Read Only External Location**: This creates a read-only external location in Unity Catalog for a given bucket as well as the corresponding AWS IAM role. + +- **Restrictive Root Bucket**: A restrictive root bucket policy can be applied to the root bucket of the workspace. **NOTE:** Please be aware this bucket is updated frequently, however, may not contain prefixes for the latest product releases. + +- **Restrictive Kinesis, STS, and S3 Endpoint Policies**: Restrictive policies for Kinesis, STS, and S3 endpoints can be added for Databricks specific assets. **NOTE:** Please be aware thse policies could be updated and may result in potentially breaking changes. If this is the case, we recommend removing the policy. + +- **System Tables**: System tables are a Databricks-hosted analytical store of your account’s operational data found in the system catalog. System tables can be used for historical observability across your account. This is currently in public preview, so is optional to enable or not. + +- **Workspace Admin. Configurations**: Workspace administration configurations that can be enabled that align with security best practices. The Terraform resource is experimental, which is why it is optional. Documentation on each configuration is provided in the Terraform file. + + +## Solution Accelerators + +- **Security Analysis Tool (SAT)**: The Security Analysis Tool analyzes customer's Databricks account and workspace security configurations and provides recommendations that can help them follow Databricks' security best practices. This can be enabled into the workspace that is being created. **NOTE:** Please be aware this creates a cluster, a job, and a dashboard within your environment. + +- **Audit Log Alerting**: Audit Log Alerting, based on this [blog post](https://www.databricks.com/blog/improve-lakehouse-security-monitoring-using-system-tables-databricks-unity-catalog), creates 40+ SQL alerts to monitor for incidents based on a Zero Trust Architecture (ZTA) model. **NOTE:** Please be aware this creates a cluster, a job, and queries within your environment. + + +## Additional Security Recommendations and Opportunities + +In this section, we break down additional security recommendations and opportunities to maintain a strong security posture that either cannot be configured into this Terraform script or is very specific to individual customers (e.g. SCIM, SSO, Front-End PrivateLink, etc.) + +- **Segment Workspaces for Various Levels of Data Separation**: While Databricks has numerous capabilities for isolating different workloads, such as table ACLs and IAM passthrough for very sensitive workloads, the primary isolation method is to move sensitive workloads to a different workspace. This sometimes happens when a customer has very different teams (for example, a security team and a marketing team) who must both analyze different data in Databricks. + +- **Avoid Storing Production Datasets in Databricks File Store**: Because the DBFS root is accessible to all users in a workspace, all users can access any data stored here. It is important to instruct users to avoid using this location for storing sensitive data. The default location for managed tables in the Hive metastore on Databricks is the DBFS root; to prevent end users who create managed tables from writing to the DBFS root, declare a location on external storage when creating databases in the Hive metastore. + +- **Single Sign-On, Multi-factor Authentication, SCIM Provisioning**: Most production or enterprise deployments enable their workspaces to use [Single Sign-On (SSO)](https://docs.databricks.com/administration-guide/users-groups/single-sign-on/index.html) and multi-factor authentication (MFA). As users are added, changed, and deleted, we recommended customers integrate [SCIM (System for Cross-domain Identity Management)](https://docs.databricks.com/dev-tools/api/latest/scim/index.html)to their account console to sync these actions. + +- **Backup Assets from the Databricks Control Plane**: While Databricks does not offer disaster recovery services, many customers use Databricks capabilities, including the Account API, to create a cold (standby) workspace in another region. This can be done using various tools such as the Databricks [migration tool](https://github.com/databrickslabs/migrate), [Databricks sync](https://github.com/databrickslabs/databricks-sync), or the [Terraform exporter](https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/experimental-exporter) + +- **Regularly Restart Databricks Clusters**: When you restart a cluster, it gets the latest images for the compute resource containers and the VM hosts. It is particularly important to schedule regular restarts for long-running clusters such as those used for processing streaming data. If you enable the compliance security profile for your account or your workspace, long-running clusters are automatically restarted after 25 days. Databricks recommends that admins restart clusters manually during a scheduled maintenance window. This reduces the risk of an auto-restart disrupting a scheduled job. + +- **Evaluate Whether your Workflow requires using Git Repos or CI/CD**: Mature organizations often build production workloads by using CI/CD to integrate code scanning, better control permissions, perform linting, and more. When there is highly sensitive data analyzed, a CI/CD process can also allow scanning for known scenarios such as hard coded secrets. + + +## Getting Started + +1. Clone this Repo +2. Install [Terraform](https://developer.hashicorp.com/terraform/downloads) +3. Decide which [operation](https://github.com/databricks/terraform-databricks-sra/tree/main/aws-gov/tf#operation-mode) mode you'd like to use. +4. Fill out `sra.tf` in place +5. Fill out `template.tfvars.example` remove the .example part of the file name +6. Configure the [AWS](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration) and [Databricks](https://registry.terraform.io/providers/databricks/databricks/latest/docs#authentication) provider authentication +7. CD into `tf` +8. Run `terraform init` +9. Run `terraform validate` +10. From `tf` directory, run `terraform plan -var-file ../example.tfvars` +11. Run `terraform apply -var-file ../example.tfvars` + + +## Network Diagram - Sandbox +![Architecture Diagram](https://github.com/databricks/terraform-databricks-sra/blob/main/aws-gov/img/Sandbox%20-%20Network%20Topology.png) + + +## Network Diagram - Firewall +![Architecture Diagram](https://github.com/databricks/terraform-databricks-sra/blob/main/aws-gov/img/Firewall%20-%20Network%20Topology.png) + + +## Network Diagram - Isolated +![Architecture Diagram](https://github.com/databricks/terraform-databricks-sra/blob/main/aws-gov/img/Isolated%20-%20Network%20Topology.png) + + +## FAQ + +- **I've cloned the GitHub repo, what's the recommended way to add Databricks additional resources to it?** + +If you'd like to add additional resources to the repository, the first step is to identify if this resource is using the **account** or **workspace** provider. + +For example, if it uses the **account** provider, then we'd recommend creating a new module under the [modules/sra/databricks_account](https://github.com/databricks/terraform-databricks-sra/tree/main/aws-gov/tf/modules/sra/databricks_account) folder. Then, that module can be called in the top level [databricks_account.tf](https://github.com/databricks/terraform-databricks-sra/blob/main/aws-gov/tf/modules/sra/databricks_account.tf) file. This process is the same for the workspace provider by placing a new module in the [modules/sra/databricks_workspace folder](https://github.com/databricks/terraform-databricks-sra/tree/main/aws-gov/tf/modules/sra/databricks_workspace) and call it in the [databricks_workspace.tf](https://github.com/databricks/terraform-databricks-sra/blob/main/aws-gov/tf/modules/sra/databricks_workspace.tf) file. \ No newline at end of file diff --git a/aws-gov/img/Firewall - Network Topology.png b/aws-gov/img/Firewall - Network Topology.png new file mode 100644 index 0000000..5c8748e Binary files /dev/null and b/aws-gov/img/Firewall - Network Topology.png differ diff --git a/aws-gov/img/Firewall - VPC Resource Map Example.png b/aws-gov/img/Firewall - VPC Resource Map Example.png new file mode 100644 index 0000000..b5ce80c Binary files /dev/null and b/aws-gov/img/Firewall - VPC Resource Map Example.png differ diff --git a/aws-gov/img/Isolated - Network Topology.png b/aws-gov/img/Isolated - Network Topology.png new file mode 100644 index 0000000..b86bbe6 Binary files /dev/null and b/aws-gov/img/Isolated - Network Topology.png differ diff --git a/aws-gov/img/Isolated - VPC Resource Map Example.png b/aws-gov/img/Isolated - VPC Resource Map Example.png new file mode 100644 index 0000000..efdb5ec Binary files /dev/null and b/aws-gov/img/Isolated - VPC Resource Map Example.png differ diff --git a/aws-gov/img/Sandbox - Network Topology.png b/aws-gov/img/Sandbox - Network Topology.png new file mode 100644 index 0000000..4e4688b Binary files /dev/null and b/aws-gov/img/Sandbox - Network Topology.png differ diff --git a/aws-gov/img/Sandbox - VPC Resource Map Example.png b/aws-gov/img/Sandbox - VPC Resource Map Example.png new file mode 100644 index 0000000..a990558 Binary files /dev/null and b/aws-gov/img/Sandbox - VPC Resource Map Example.png differ diff --git a/aws-gov/tf/modules/sra/cmk.tf b/aws-gov/tf/modules/sra/cmk.tf new file mode 100644 index 0000000..6405a36 --- /dev/null +++ b/aws-gov/tf/modules/sra/cmk.tf @@ -0,0 +1,123 @@ +// EXPLANATION: The customer-managed keys for workspace and managed storage + +locals { + cmk_admin_value = var.cmk_admin_arn == null ? "arn:aws-us-gov:iam::${var.aws_account_id}:root" : var.cmk_admin_arn +} + +resource "aws_kms_key" "workspace_storage" { + description = "KMS key for databricks workspace storage" + policy = jsonencode({ + Version : "2012-10-17", + "Id" : "key-policy-workspace-storage", + Statement : [ + { + "Sid" : "Enable IAM User Permissions", + "Effect" : "Allow", + "Principal" : { + "AWS" : [local.cmk_admin_value] + }, + "Action" : "kms:*", + "Resource" : "*" + }, + { + "Sid" : "Allow Databricks to use KMS key for DBFS", + "Effect" : "Allow", + "Principal" : { + "AWS" : "arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root" + }, + "Action" : [ + "kms:Encrypt", + "kms:Decrypt", + "kms:ReEncrypt*", + "kms:GenerateDataKey*", + "kms:DescribeKey" + ], + "Resource" : "*", + "Condition" : { + "StringEquals" : { + "aws:PrincipalTag/DatabricksAccountId" : "${var.databricks_account_id}" + } + } + }, + { + "Sid" : "Allow Databricks to use KMS key for EBS", + "Effect" : "Allow", + "Principal" : { + "AWS" : "${aws_iam_role.cross_account_role.arn}" + }, + "Action" : [ + "kms:Decrypt", + "kms:GenerateDataKey*", + "kms:CreateGrant", + "kms:DescribeKey" + ], + "Resource" : "*", + "Condition" : { + "ForAnyValue:StringLike" : { + "kms:ViaService" : "ec2.*.amazonaws.com" + } + } + } + ] + }) + depends_on = [aws_iam_role.cross_account_role] + + tags = { + Name = "${var.resource_prefix}-workspace-storage-key" + Project = var.resource_prefix + } +} + + +resource "aws_kms_alias" "workspace_storage_key_alias" { + name = "alias/${var.resource_prefix}-workspace-storage-key" + target_key_id = aws_kms_key.workspace_storage.id +} + +## CMK for Managed Storage + +resource "aws_kms_key" "managed_storage" { + description = "KMS key for managed storage" + policy = jsonencode({ Version : "2012-10-17", + "Id" : "key-policy-managed-storage", + Statement : [ + { + "Sid" : "Enable IAM User Permissions", + "Effect" : "Allow", + "Principal" : { + "AWS" : [local.cmk_admin_value] + }, + "Action" : "kms:*", + "Resource" : "*" + }, + { + "Sid" : "Allow Databricks to use KMS key for managed services in the control plane", + "Effect" : "Allow", + "Principal" : { + "AWS" : "arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root" + }, + "Action" : [ + "kms:Encrypt", + "kms:Decrypt" + ], + "Resource" : "*", + "Condition" : { + "StringEquals" : { + "aws:PrincipalTag/DatabricksAccountId" : ["${var.databricks_account_id}"] + } + } + } + ] + } + ) + + tags = { + Project = var.resource_prefix + Name = "${var.resource_prefix}-managed-storage-key" + } +} + +resource "aws_kms_alias" "managed_storage_key_alias" { + name = "alias/${var.resource_prefix}-managed-storage-key" + target_key_id = aws_kms_key.managed_storage.key_id +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/credential.tf b/aws-gov/tf/modules/sra/credential.tf new file mode 100644 index 0000000..476a0be --- /dev/null +++ b/aws-gov/tf/modules/sra/credential.tf @@ -0,0 +1,231 @@ +// EXPLANATION: The cross-account role for the Databricks workspace + +// Cross Account Trust Policy +data "aws_iam_policy_document" "passrole_for_cross_account_credential" { + statement { + effect = "Allow" + actions = ["sts:AssumeRole"] + principals { + identifiers = ["arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root"] + type = "AWS" + } + condition { + test = "StringEquals" + variable = "sts:ExternalId" + values = [var.databricks_account_id] + } + } +} + +// Cross Account Role +resource "aws_iam_role" "cross_account_role" { + name = "${var.resource_prefix}-cross-account" + assume_role_policy = data.aws_iam_policy_document.passrole_for_cross_account_credential.json + tags = { + Name = "${var.resource_prefix}-cross-account" + Project = var.resource_prefix + } +} + +resource "aws_iam_role_policy" "cross_account" { + name = "${var.resource_prefix}-crossaccount-policy" + role = aws_iam_role.cross_account_role.id + policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Sid" : "NonResourceBasedPermissions", + "Effect" : "Allow", + "Action" : [ + "ec2:CancelSpotInstanceRequests", + "ec2:DescribeAvailabilityZones", + "ec2:DescribeIamInstanceProfileAssociations", + "ec2:DescribeInstanceStatus", + "ec2:DescribeInstances", + "ec2:DescribeInternetGateways", + "ec2:DescribeNatGateways", + "ec2:DescribeNetworkAcls", + "ec2:DescribePrefixLists", + "ec2:DescribeReservedInstancesOfferings", + "ec2:DescribeRouteTables", + "ec2:DescribeSecurityGroups", + "ec2:DescribeSpotInstanceRequests", + "ec2:DescribeSpotPriceHistory", + "ec2:DescribeSubnets", + "ec2:DescribeVolumes", + "ec2:DescribeVpcAttribute", + "ec2:DescribeVpcs", + "ec2:CreateTags", + "ec2:DeleteTags", + "ec2:RequestSpotInstances" + ], + "Resource" : [ + "*" + ] + }, + { + "Sid" : "InstancePoolsSupport", + "Effect" : "Allow", + "Action" : [ + "ec2:AssociateIamInstanceProfile", + "ec2:DisassociateIamInstanceProfile", + "ec2:ReplaceIamInstanceProfileAssociation" + ], + "Resource" : "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:instance/*", + "Condition" : { + "StringEquals" : { + "ec2:ResourceTag/Vendor" : "Databricks" + } + } + }, + { + "Sid" : "AllowEc2RunInstancePerTag", + "Effect" : "Allow", + "Action" : "ec2:RunInstances", + "Resource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:volume/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:instance/*" + ], + "Condition" : { + "StringEquals" : { + "aws:RequestTag/Vendor" : "Databricks" + } } + }, + { + "Sid" : "AllowEc2RunInstancePerVPCid", + "Effect" : "Allow", + "Action" : "ec2:RunInstances", + "Resource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:network-interface/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:subnet/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:security-group/*" + ], + "Condition" : { + "StringEquals" : { + "ec2:vpc" : "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:vpc/${var.custom_vpc_id != null ? var.custom_vpc_id : module.vpc[0].vpc_id}" + } + } + }, + { + "Sid" : "AllowEc2RunInstanceOtherResources", + "Effect" : "Allow", + "Action" : "ec2:RunInstances", + "NotResource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:network-interface/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:subnet/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:security-group/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:volume/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:instance/*" + ] + }, + { + "Sid" : "DatabricksSuppliedImages", + "Effect" : "Deny", + "Action" : "ec2:RunInstances", + "Resource" : [ + "arn:aws-us-gov:ec2:*:*:image/*" + ], + "Condition" : { + "StringNotEquals" : { + "ec2:Owner" : "044732911619" + } + } + }, + { + "Sid" : "EC2TerminateInstancesTag", + "Effect" : "Allow", + "Action" : [ + "ec2:TerminateInstances" + ], + "Resource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:instance/*" + ], + "Condition" : { + "StringEquals" : { + "ec2:ResourceTag/Vendor" : "Databricks" + } + } + }, + { + "Sid" : "EC2AttachDetachVolumeTag", + "Effect" : "Allow", + "Action" : [ + "ec2:AttachVolume", + "ec2:DetachVolume" + ], + "Resource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:instance/*", + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:volume/*" + ], + "Condition" : { + "StringEquals" : { + "ec2:ResourceTag/Vendor" : "Databricks" + } + } + }, + { + "Sid" : "EC2CreateVolumeByTag", + "Effect" : "Allow", + "Action" : [ + "ec2:CreateVolume" + ], + "Resource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:volume/*" + ], + "Condition" : { + "StringEquals" : { + "aws:RequestTag/Vendor" : "Databricks" + } + } + }, + { + "Sid" : "EC2DeleteVolumeByTag", + "Effect" : "Allow", + "Action" : [ + "ec2:DeleteVolume" + ], + "Resource" : [ + "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:volume/*" + ], + "Condition" : { + "StringEquals" : { + "ec2:ResourceTag/Vendor" : "Databricks" + } + } + }, + { + "Effect" : "Allow", + "Action" : [ + "iam:CreateServiceLinkedRole", + "iam:PutRolePolicy" + ], + "Resource" : "arn:aws-us-gov:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot", + "Condition" : { + "StringLike" : { + "iam:AWSServiceName" : "spot.amazonaws.com" + } + } + }, + { + "Sid" : "VpcNonresourceSpecificActions", + "Effect" : "Allow", + "Action" : [ + "ec2:AuthorizeSecurityGroupEgress", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:RevokeSecurityGroupEgress", + "ec2:RevokeSecurityGroupIngress" + ], + "Resource" : "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:security-group/${var.custom_sg_id != null ? var.custom_sg_id : aws_security_group.sg[0].id}", + "Condition" : { + "StringEquals" : { + "ec2:vpc" : "arn:aws-us-gov:ec2:${var.region}:${var.aws_account_id}:vpc/${var.custom_vpc_id != null ? var.custom_vpc_id : module.vpc[0].vpc_id}" + } + } + } + ] + } + ) + depends_on = [ + module.vpc, aws_security_group.sg + ] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening.tf b/aws-gov/tf/modules/sra/data_plane_hardening.tf new file mode 100644 index 0000000..cbb2cda --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening.tf @@ -0,0 +1,42 @@ +// EXPLANATION: Optional modules that harden the AWS data plane + +// Implement an AWS Firewall +module "harden_firewall" { + count = var.operation_mode == "firewall" ? 1 : 0 + source = "./data_plane_hardening/firewall" + providers = { + aws = aws + } + + vpc_id = module.vpc[0].vpc_id + vpc_cidr_range = var.vpc_cidr_range + public_subnets_cidr = var.public_subnets_cidr + private_subnets_cidr = module.vpc[0].private_subnets_cidr_blocks + private_subnet_rt = module.vpc[0].private_route_table_ids + firewall_subnets_cidr = var.firewall_subnets_cidr + firewall_allow_list = var.firewall_allow_list + hive_metastore_fqdn = var.hms_fqdn[var.databricks_gov_shard] + availability_zones = var.availability_zones + region = var.region + resource_prefix = var.resource_prefix + + depends_on = [module.databricks_mws_workspace] +} + + +// Restrictive DBFS bucket policy +module "restrictive_root_bucket" { + count = var.enable_restrictive_root_bucket_boolean ? 1 : 0 + source = "./data_plane_hardening/restrictive_root_bucket" + providers = { + aws = aws + } + + workspace_id = module.databricks_mws_workspace.workspace_id + region_name = var.region_name + root_s3_bucket = "${var.resource_prefix}-workspace-root-storage" + databricks_gov_shard = var.databricks_gov_shard + databricks_prod_aws_account_id = var.databricks_prod_aws_account_id + + depends_on = [module.databricks_mws_workspace] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening/firewall/firewall.tf b/aws-gov/tf/modules/sra/data_plane_hardening/firewall/firewall.tf new file mode 100644 index 0000000..18536c0 --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening/firewall/firewall.tf @@ -0,0 +1,289 @@ +// EXPLANATION: Creates an egress firewall around the dataplane + +// Public Subnet +resource "aws_subnet" "public" { + vpc_id = var.vpc_id + count = length(var.public_subnets_cidr) + cidr_block = element(var.public_subnets_cidr, count.index) + availability_zone = element(var.availability_zones, count.index) + map_public_ip_on_launch = true + tags = { + Name = "${var.resource_prefix}-public-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix + } +} + +// EIP +resource "aws_eip" "ngw_eip" { + count = length(var.public_subnets_cidr) + domain = "vpc" +} + +// NGW +resource "aws_nat_gateway" "ngw" { + count = length(var.public_subnets_cidr) + allocation_id = element(aws_eip.ngw_eip.*.id, count.index) + subnet_id = element(aws_subnet.public.*.id, count.index) + depends_on = [aws_internet_gateway.igw] + tags = { + Name = "${var.resource_prefix}-ngw-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix + } +} + +// Private Subnet Route +resource "aws_route" "private" { + count = length(var.private_subnets_cidr) + route_table_id = element(var.private_subnet_rt, count.index) + destination_cidr_block = "0.0.0.0/0" + nat_gateway_id = element(aws_nat_gateway.ngw.*.id, count.index) +} + +// Public RT +resource "aws_route_table" "public_rt" { + count = length(var.public_subnets_cidr) + vpc_id = var.vpc_id + tags = { + Name = "${var.resource_prefix}-public-rt-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix + } +} + +// Public RT Associations +resource "aws_route_table_association" "public" { + count = length(var.public_subnets_cidr) + subnet_id = element(aws_subnet.public.*.id, count.index) + route_table_id = element(aws_route_table.public_rt.*.id, count.index) + depends_on = [aws_subnet.public] +} + +// Firewall Subnet +resource "aws_subnet" "firewall" { + vpc_id = var.vpc_id + count = length(var.firewall_subnets_cidr) + cidr_block = element(var.firewall_subnets_cidr, count.index) + availability_zone = element(var.availability_zones, count.index) + map_public_ip_on_launch = false + tags = { + Name = "${var.resource_prefix}-firewall-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix + } +} + +// Firewall RT +resource "aws_route_table" "firewall_rt" { + count = length(var.firewall_subnets_cidr) + vpc_id = var.vpc_id + tags = { + Name = "${var.resource_prefix}-firewall-rt-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix + } +} + +// Firewall RT Associations +resource "aws_route_table_association" "firewall" { + count = length(var.firewall_subnets_cidr) + subnet_id = element(aws_subnet.firewall.*.id, count.index) + route_table_id = element(aws_route_table.firewall_rt.*.id, count.index) +} + +// IGW +resource "aws_internet_gateway" "igw" { + vpc_id = var.vpc_id + tags = { + Name = "${var.resource_prefix}-igw" + Project = var.resource_prefix + } +} + +// IGW RT +resource "aws_route_table" "igw_rt" { + vpc_id = var.vpc_id + tags = { + Name = "${var.resource_prefix}-igw-rt" + Project = var.resource_prefix + } +} + +// IGW RT Associations +resource "aws_route_table_association" "igw" { + gateway_id = aws_internet_gateway.igw.id + route_table_id = aws_route_table.igw_rt.id +} + +// Local Map for Availability Zone to Index +locals { + az_to_index_map = { + for idx, az in var.availability_zones : + az => idx + } + + firewall_endpoints_by_az = { + for sync_state in aws_networkfirewall_firewall.nfw.firewall_status[0].sync_states : + sync_state.availability_zone => sync_state.attachment[0].endpoint_id + } + + az_to_endpoint_map = { + for az in var.availability_zones : + az => lookup(local.firewall_endpoints_by_az, az, null) + } +} + +// Public Route +resource "aws_route" "public" { + for_each = local.az_to_endpoint_map + route_table_id = aws_route_table.public_rt[local.az_to_index_map[each.key]].id + destination_cidr_block = "0.0.0.0/0" + vpc_endpoint_id = each.value + depends_on = [aws_networkfirewall_firewall.nfw] +} + +// Firewall Outbound Route +resource "aws_route" "firewall_outbound" { + count = length(var.firewall_subnets_cidr) + route_table_id = element(aws_route_table.firewall_rt.*.id, count.index) + destination_cidr_block = "0.0.0.0/0" + gateway_id = aws_internet_gateway.igw.id +} + +// Firewall Inbound Route +resource "aws_route" "firewall_inbound" { + for_each = local.az_to_endpoint_map + route_table_id = aws_route_table.igw_rt.id + destination_cidr_block = element(var.public_subnets_cidr, index(var.availability_zones, each.key)) + vpc_endpoint_id = each.value + depends_on = [aws_networkfirewall_firewall.nfw] +} + +// FQDN Allow List +resource "aws_networkfirewall_rule_group" "databricks_fqdn_allowlist" { + capacity = 100 + name = "${var.resource_prefix}-${var.region}-databricks-fqdn-allowlist" + type = "STATEFUL" + rule_group { + stateful_rule_options { + rule_order = "STRICT_ORDER" + } + rules_source { + rules_source_list { + generated_rules_type = "ALLOWLIST" + target_types = ["TLS_SNI", "HTTP_HOST"] + targets = var.firewall_allow_list + } + } + rule_variables { + ip_sets { + key = "HOME_NET" + ip_set { + definition = [var.vpc_cidr_range] + } + } + } + } + tags = { + Name = "${var.resource_prefix}-${var.region}-databricks-fqdn-allowlist" + Project = var.resource_prefix + } +} + +data "dns_a_record_set" "metastore_dns" { + host = var.hive_metastore_fqdn +} + +// JDBC Firewall group IP allow list +resource "aws_networkfirewall_rule_group" "databricks_metastore_allowlist" { + capacity = 100 + name = "${var.resource_prefix}-${var.region}-databricks-metastore-allowlist" + type = "STATEFUL" + rule_group { + stateful_rule_options { + rule_order = "STRICT_ORDER" + } + rules_source { + dynamic "stateful_rule" { + for_each = toset(data.dns_a_record_set.metastore_dns.addrs) + content { + action = "PASS" + header { + destination = stateful_rule.value + destination_port = 3306 + direction = "FORWARD" + protocol = "TCP" + source = "ANY" + source_port = "ANY" + } + rule_option { + keyword = "sid" + settings = ["1"] + } + } + } + stateful_rule { + action = "DROP" + header { + destination = "0.0.0.0/0" + destination_port = 3306 + direction = "FORWARD" + protocol = "TCP" + source = "ANY" + source_port = "ANY" + } + rule_option { + keyword = "sid" + settings = ["2"] + } + } + } + } + tags = { + Name = "${var.resource_prefix}-${var.region}-databricks-metastore-allowlist" + Project = var.resource_prefix + } +} + +// Firewall policy +resource "aws_networkfirewall_firewall_policy" "databricks_nfw_policy" { + name = "${var.resource_prefix}-firewall-policy" + + firewall_policy { + + stateful_engine_options { + rule_order = "STRICT_ORDER" + } + stateless_default_actions = ["aws:forward_to_sfe"] + stateless_fragment_default_actions = ["aws:forward_to_sfe"] + stateful_default_actions = ["aws:drop_established"] + + stateful_rule_group_reference { + priority = 1 + resource_arn = aws_networkfirewall_rule_group.databricks_fqdn_allowlist.arn + } + + stateful_rule_group_reference { + priority = 2 + resource_arn = aws_networkfirewall_rule_group.databricks_metastore_allowlist.arn + } + } + + tags = { + Name = "${var.resource_prefix}-firewall-policy" + Project = var.resource_prefix + } +} + +// Firewall +resource "aws_networkfirewall_firewall" "nfw" { + name = "${var.resource_prefix}-nfw" + firewall_policy_arn = aws_networkfirewall_firewall_policy.databricks_nfw_policy.arn + vpc_id = var.vpc_id + dynamic "subnet_mapping" { + for_each = aws_subnet.firewall[*].id + content { + subnet_id = subnet_mapping.value + } + } + tags = { + Name = "${var.resource_prefix}-${var.region}-databricks-nfw" + Project = var.resource_prefix + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening/firewall/provider.tf b/aws-gov/tf/modules/sra/data_plane_hardening/firewall/provider.tf new file mode 100644 index 0000000..7617f6b --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening/firewall/provider.tf @@ -0,0 +1,10 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + } + dns = { + source = "hashicorp/dns" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening/firewall/variables.tf b/aws-gov/tf/modules/sra/data_plane_hardening/firewall/variables.tf new file mode 100644 index 0000000..2d8b5b8 --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening/firewall/variables.tf @@ -0,0 +1,43 @@ +variable "availability_zones" { + type = list(string) +} + +variable "firewall_allow_list" { + type = list(string) +} + +variable "firewall_subnets_cidr" { + type = list(string) +} + +variable "hive_metastore_fqdn" { + type = string +} + +variable "private_subnet_rt" { + type = list(string) +} + +variable "private_subnets_cidr" { + type = list(string) +} + +variable "public_subnets_cidr" { + type = list(string) +} + +variable "region" { + type = string +} + +variable "resource_prefix" { + type = string +} + +variable "vpc_cidr_range" { + type = string +} + +variable "vpc_id" { + type = string +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/provider.tf b/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/provider.tf new file mode 100644 index 0000000..7afdcf4 --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/restrictive_root_bucket.tf b/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/restrictive_root_bucket.tf new file mode 100644 index 0000000..4e3fafc --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/restrictive_root_bucket.tf @@ -0,0 +1,71 @@ +// EXPLANATION: Creates a restrictive root bucket policy + +// Restrictive Bucket Policy +resource "aws_s3_bucket_policy" "databricks_bucket_restrictive_policy" { + bucket = var.root_s3_bucket + policy = jsonencode({ + Version = "2012-10-17", + Statement = [ + { + Sid = "Grant Databricks Read Access", + Effect = "Allow", + Principal = { + AWS = "arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root" + }, + Action = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:ListBucket", + "s3:GetBucketLocation" + ], + Resource = [ + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}" + ] + }, + { + Sid = "Grant Databricks Write Access", + Effect = "Allow", + Principal = { + AWS = "arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root" + }, + Action = [ + "s3:PutObject", + "s3:DeleteObject" + ], + Resource = [ + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/0_databricks_dev", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/ephemeral/${var.region_name}-prod/${var.workspace_id}/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}.*/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/databricks/init/*/*.sh", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/user/hive/warehouse/*.db/", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/user/hive/warehouse/*.db/*-*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/user/hive/warehouse/*__PLACEHOLDER__/", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/user/hive/warehouse/*.db/*__PLACEHOLDER__/", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/FileStore/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/databricks/mlflow/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/databricks/mlflow-*/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/mlflow-*/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/pipelines/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/local_disk0/tmp/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/${var.region_name}-prod/${var.workspace_id}/tmp/*" + ] + }, + { + Sid = "AllowSSLRequestsOnly", + Effect = "Deny", + Action = ["s3:*"], + Principal = "*", + Resource = [ + "arn:aws-us-gov:s3:::${var.root_s3_bucket}/*", + "arn:aws-us-gov:s3:::${var.root_s3_bucket}" + ], + Condition = { + Bool = { + "aws:SecureTransport" = "false" + } + } + } + ] + }) +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/variables.tf b/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/variables.tf new file mode 100644 index 0000000..a48b064 --- /dev/null +++ b/aws-gov/tf/modules/sra/data_plane_hardening/restrictive_root_bucket/variables.tf @@ -0,0 +1,18 @@ +variable "region_name" { + type = string +} + +variable "root_s3_bucket" { + type = string +} + +variable "workspace_id" { + type = string +} + +variable "databricks_gov_shard" { + type = string +} +variable "databricks_prod_aws_account_id" { + type = map(string) +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account.tf b/aws-gov/tf/modules/sra/databricks_account.tf new file mode 100644 index 0000000..bc03e38 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account.tf @@ -0,0 +1,80 @@ +// EXPLANATION: All modules that reside at the account level + +// Billable Usage and Audit Logs +module "log_delivery" { + source = "./databricks_account/logging_configuration" + count = var.enable_logging_boolean ? 1 : 0 + providers = { + databricks = databricks.mws + } + + databricks_account_id = var.databricks_account_id + resource_prefix = var.resource_prefix + databricks_gov_shard = var.databricks_gov_shard + databricks_prod_aws_account_id = var.databricks_prod_aws_account_id + log_delivery_role_name = var.log_delivery_role_name +} + + +// Create Unity Catalog Metastore - No Root Storage +module "uc_init" { + source = "./databricks_account/uc_init" + providers = { + databricks = databricks.mws + } + + aws_account_id = var.aws_account_id + databricks_account_id = var.databricks_account_id + resource_prefix = var.resource_prefix + region = var.region + metastore_name = join("", [var.resource_prefix, "-", var.region, "-", "uc"]) + metastore_exists = var.metastore_exists +} + +// Unity Catalog Assignment +module "uc_assignment" { + source = "./databricks_account/uc_assignment" + providers = { + databricks = databricks.mws + } + + metastore_id = module.uc_init.metastore_id + region = var.region + workspace_id = module.databricks_mws_workspace.workspace_id + depends_on = [module.databricks_mws_workspace, module.uc_init] +} + +// Create Databricks Workspace +module "databricks_mws_workspace" { + source = "./databricks_account/workspace" + providers = { + databricks = databricks.mws + } + + databricks_account_id = var.databricks_account_id + resource_prefix = var.resource_prefix + security_group_ids = var.custom_sg_id != null ? [var.custom_sg_id] : [aws_security_group.sg[0].id] + subnet_ids = var.custom_private_subnet_ids != null ? var.custom_private_subnet_ids : module.vpc[0].private_subnets + vpc_id = var.custom_vpc_id != null ? var.custom_vpc_id : module.vpc[0].vpc_id + cross_account_role_arn = aws_iam_role.cross_account_role.arn + bucket_name = aws_s3_bucket.root_storage_bucket.id + region = var.region + backend_rest = var.custom_workspace_vpce_id != null ? var.custom_workspace_vpce_id : aws_vpc_endpoint.backend_rest[0].id + backend_relay = var.custom_relay_vpce_id != null ? var.custom_relay_vpce_id : aws_vpc_endpoint.backend_relay[0].id + managed_storage_key = aws_kms_key.managed_storage.arn + workspace_storage_key = aws_kms_key.workspace_storage.arn + managed_storage_key_alias = aws_kms_alias.managed_storage_key_alias.name + workspace_storage_key_alias = aws_kms_alias.workspace_storage_key_alias.name +} + +// User Workspace Assignment (Admin) +module "user_assignment" { + source = "./databricks_account/user_assignment" + providers = { + databricks = databricks.mws + } + + created_workspace_id = module.databricks_mws_workspace.workspace_id + workspace_access = var.user_workspace_admin + depends_on = [module.uc_assignment, module.databricks_mws_workspace] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf b/aws-gov/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf new file mode 100644 index 0000000..cea829f --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf @@ -0,0 +1,163 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_log_delivery + +// S3 Log Bucket +resource "aws_s3_bucket" "log_delivery" { + bucket = "${var.resource_prefix}-log-delivery" + force_destroy = true + tags = { + Name = "${var.resource_prefix}-log-delivery" + Project = var.resource_prefix + } +} + +// S3 Bucket Versioning +resource "aws_s3_bucket_versioning" "log_delivery" { + bucket = aws_s3_bucket.log_delivery.id + versioning_configuration { + status = "Disabled" + } +} + +// S3 Public Access Block +resource "aws_s3_bucket_public_access_block" "log_delivery" { + bucket = aws_s3_bucket.log_delivery.id + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true + depends_on = [aws_s3_bucket.log_delivery] +} + +// S3 Policy for Log Delivery Data +data "databricks_aws_bucket_policy" "log_delivery" { + full_access_role = aws_iam_role.log_delivery.arn + bucket = aws_s3_bucket.log_delivery.bucket +} + +// S3 Policy for Log Delivery Resources +resource "aws_s3_bucket_policy" "log_delivery" { + bucket = aws_s3_bucket.log_delivery.id + policy = jsonencode({ + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow", + "Principal" : { + "AWS" : ["${aws_iam_role.log_delivery.arn}"] + }, + "Action" : "s3:GetBucketLocation", + "Resource" : "arn:aws-us-gov:s3:::${var.resource_prefix}-log-delivery" + }, + { + "Effect" : "Allow", + "Principal" : { + "AWS" : ["${aws_iam_role.log_delivery.arn}"] + }, + "Action" : [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:PutObjectAcl", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts" + ], + "Resource" : [ + "arn:aws-us-gov:s3:::${var.resource_prefix}-log-delivery", + "arn:aws-us-gov:s3:::${var.resource_prefix}-log-delivery/*" + ] + }, + { + "Effect" : "Allow", + "Principal" : { + "AWS" : ["${aws_iam_role.log_delivery.arn}"] + }, + "Action" : "s3:ListBucket", + "Resource" : "arn:aws-us-gov:s3:::${var.resource_prefix}-log-delivery" + } + ] + } + ) + depends_on = [ + aws_s3_bucket.log_delivery + ] +} + +// IAM Role + +// Assume Role Policy Log Delivery +data "aws_iam_policy_document" "passrole_for_log_delivery" { + statement { + effect = "Allow" + actions = ["sts:AssumeRole"] + principals { + identifiers = ["arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:${var.log_delivery_role_name[var.databricks_gov_shard]}"] + type = "AWS" + } + condition { + test = "StringEquals" + variable = "sts:ExternalId" + values = [var.databricks_account_id] + } + } +} + +// Log Delivery IAM Role +resource "aws_iam_role" "log_delivery" { + name = "${var.resource_prefix}-log-delivery" + description = "(${var.resource_prefix}) Log Delivery Role" + assume_role_policy = data.aws_iam_policy_document.passrole_for_log_delivery.json + tags = { + Name = "${var.resource_prefix}-log-delivery-role" + Project = var.resource_prefix + } +} + +// Databricks Configurations + +// Databricks Credential Configuration for Logs +resource "databricks_mws_credentials" "log_writer" { + account_id = var.databricks_account_id + credentials_name = "${var.resource_prefix}-log-delivery-credential" + role_arn = aws_iam_role.log_delivery.arn + depends_on = [ + aws_s3_bucket_policy.log_delivery + ] +} + +// Databricks Storage Configuration for Logs +resource "databricks_mws_storage_configurations" "log_bucket" { + account_id = var.databricks_account_id + storage_configuration_name = "${var.resource_prefix}-log-delivery-bucket" + bucket_name = aws_s3_bucket.log_delivery.bucket + depends_on = [ + aws_s3_bucket_policy.log_delivery + ] +} + +// Databricks Billable Usage Logs Configurations +resource "databricks_mws_log_delivery" "billable_usage_logs" { + account_id = var.databricks_account_id + credentials_id = databricks_mws_credentials.log_writer.credentials_id + storage_configuration_id = databricks_mws_storage_configurations.log_bucket.storage_configuration_id + delivery_path_prefix = "billable-usage-logs" + config_name = "Billable Usage Logs" + log_type = "BILLABLE_USAGE" + output_format = "CSV" + depends_on = [ + aws_s3_bucket_policy.log_delivery + ] +} + +// Databricks Audit Logs Configurations +resource "databricks_mws_log_delivery" "audit_logs" { + account_id = var.databricks_account_id + credentials_id = databricks_mws_credentials.log_writer.credentials_id + storage_configuration_id = databricks_mws_storage_configurations.log_bucket.storage_configuration_id + delivery_path_prefix = "audit-logs" + config_name = "Audit Logs" + log_type = "AUDIT_LOGS" + output_format = "JSON" + depends_on = [ + aws_s3_bucket_policy.log_delivery + ] +} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/service_principal/provider.tf b/aws-gov/tf/modules/sra/databricks_account/logging_configuration/provider.tf similarity index 100% rename from aws/tf/modules/sra/databricks_account/service_principal/provider.tf rename to aws-gov/tf/modules/sra/databricks_account/logging_configuration/provider.tf diff --git a/aws-gov/tf/modules/sra/databricks_account/logging_configuration/variables.tf b/aws-gov/tf/modules/sra/databricks_account/logging_configuration/variables.tf new file mode 100644 index 0000000..ed32d78 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/logging_configuration/variables.tf @@ -0,0 +1,19 @@ +variable "databricks_account_id" { + type = string +} + +variable "databricks_gov_shard" { + type = string +} + +variable "databricks_prod_aws_account_id" { + type = map(string) +} + +variable "log_delivery_role_name" { + type = map(string) +} + +variable "resource_prefix" { + type = string +} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/provider.tf b/aws-gov/tf/modules/sra/databricks_account/uc_assignment/provider.tf similarity index 100% rename from aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/provider.tf rename to aws-gov/tf/modules/sra/databricks_account/uc_assignment/provider.tf diff --git a/aws-gov/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf b/aws-gov/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf new file mode 100644 index 0000000..5ead29d --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf @@ -0,0 +1,6 @@ +// Metastore Assignment + +resource "databricks_metastore_assignment" "default_metastore" { + workspace_id = var.workspace_id + metastore_id = var.metastore_id +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/uc_assignment/variables.tf b/aws-gov/tf/modules/sra/databricks_account/uc_assignment/variables.tf new file mode 100644 index 0000000..8c922ed --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/uc_assignment/variables.tf @@ -0,0 +1,11 @@ +variable "metastore_id" { + type = string +} + +variable "region" { + type = string +} + +variable "workspace_id" { + type = string +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/uc_init/outputs.tf b/aws-gov/tf/modules/sra/databricks_account/uc_init/outputs.tf new file mode 100644 index 0000000..c122a8c --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/uc_init/outputs.tf @@ -0,0 +1,3 @@ +output "metastore_id" { + value = var.metastore_exists ? data.databricks_metastore.this[0].id : databricks_metastore.this[0].id +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/uc_init/provider.tf b/aws-gov/tf/modules/sra/databricks_account/uc_init/provider.tf new file mode 100644 index 0000000..72b6ed6 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/uc_init/provider.tf @@ -0,0 +1,10 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + aws = { + source = "hashicorp/aws" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/uc_init/uc_init.tf b/aws-gov/tf/modules/sra/databricks_account/uc_init/uc_init.tf new file mode 100644 index 0000000..34df58d --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/uc_init/uc_init.tf @@ -0,0 +1,14 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog + +// Optional data source - only run if the metastore exists +data "databricks_metastore" "this" { + count = var.metastore_exists ? 1 : 0 + region = var.region +} + +resource "databricks_metastore" "this" { + count = var.metastore_exists ? 0 : 1 + name = "${var.resource_prefix}-${var.region}-unity-catalog" + region = var.region + force_destroy = true +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/uc_init/variables.tf b/aws-gov/tf/modules/sra/databricks_account/uc_init/variables.tf new file mode 100644 index 0000000..ec1a35d --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/uc_init/variables.tf @@ -0,0 +1,23 @@ +variable "aws_account_id" { + type = string +} + +variable "databricks_account_id" { + type = string +} + +variable "metastore_exists" { + type = string +} + +variable "metastore_name" { + type = string +} + +variable "region" { + type = string +} + +variable "resource_prefix" { + type = string +} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/token_management/provider.tf b/aws-gov/tf/modules/sra/databricks_account/user_assignment/provider.tf similarity index 100% rename from aws/tf/modules/sra/databricks_workspace/workspace_security_modules/token_management/provider.tf rename to aws-gov/tf/modules/sra/databricks_account/user_assignment/provider.tf diff --git a/aws-gov/tf/modules/sra/databricks_account/user_assignment/user_assignment.tf b/aws-gov/tf/modules/sra/databricks_account/user_assignment/user_assignment.tf new file mode 100644 index 0000000..066c00f --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/user_assignment/user_assignment.tf @@ -0,0 +1,11 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/service_principal + +data "databricks_user" "workspace_access" { + user_name = var.workspace_access +} + +resource "databricks_mws_permission_assignment" "workspace_access" { + workspace_id = var.created_workspace_id + principal_id = data.databricks_user.workspace_access.id + permissions = ["ADMIN"] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/user_assignment/variables.tf b/aws-gov/tf/modules/sra/databricks_account/user_assignment/variables.tf new file mode 100644 index 0000000..f062416 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/user_assignment/variables.tf @@ -0,0 +1,7 @@ +variable "created_workspace_id" { + type = string +} + +variable "workspace_access" { + type = string +} diff --git a/aws-gov/tf/modules/sra/databricks_account/workspace/outputs.tf b/aws-gov/tf/modules/sra/databricks_account/workspace/outputs.tf new file mode 100644 index 0000000..066bdef --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/workspace/outputs.tf @@ -0,0 +1,7 @@ +output "workspace_url" { + value = databricks_mws_workspaces.this.workspace_url +} + +output "workspace_id" { + value = databricks_mws_workspaces.this.workspace_id +} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/public_preview/system_schema/provider.tf b/aws-gov/tf/modules/sra/databricks_account/workspace/provider.tf similarity index 100% rename from aws/tf/modules/sra/databricks_workspace/public_preview/system_schema/provider.tf rename to aws-gov/tf/modules/sra/databricks_account/workspace/provider.tf diff --git a/aws-gov/tf/modules/sra/databricks_account/workspace/variables.tf b/aws-gov/tf/modules/sra/databricks_account/workspace/variables.tf new file mode 100644 index 0000000..07748d4 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/workspace/variables.tf @@ -0,0 +1,55 @@ +variable "backend_relay" { + type = string +} + +variable "backend_rest" { + type = string +} + +variable "bucket_name" { + type = string +} + +variable "cross_account_role_arn" { + type = string +} + +variable "databricks_account_id" { + type = string +} + +variable "managed_storage_key" { + type = string +} + +variable "managed_storage_key_alias" { + type = string +} + +variable "region" { + type = string +} + +variable "resource_prefix" { + type = string +} + +variable "security_group_ids" { + type = list(string) +} + +variable "subnet_ids" { + type = list(string) +} + +variable "vpc_id" { + type = string +} + +variable "workspace_storage_key" { + type = string +} + +variable "workspace_storage_key_alias" { + type = string +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_account/workspace/workspace.tf b/aws-gov/tf/modules/sra/databricks_account/workspace/workspace.tf new file mode 100644 index 0000000..185be7d --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_account/workspace/workspace.tf @@ -0,0 +1,99 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces + + +// Wait on Credential Due to Race Condition +// https://kb.databricks.com/en_US/terraform/failed-credential-validation-checks-error-with-terraform +resource "null_resource" "previous" {} + +resource "time_sleep" "wait_30_seconds" { + depends_on = [null_resource.previous] + + create_duration = "30s" +} + +// Credential Configuration +resource "databricks_mws_credentials" "this" { + role_arn = var.cross_account_role_arn + credentials_name = "${var.resource_prefix}-credentials" + depends_on = [time_sleep.wait_30_seconds] +} + +// Storage Configuration +resource "databricks_mws_storage_configurations" "this" { + account_id = var.databricks_account_id + bucket_name = var.bucket_name + storage_configuration_name = "${var.resource_prefix}-storage" +} + +// Backend REST VPC Endpoint Configuration +resource "databricks_mws_vpc_endpoint" "backend_rest" { + account_id = var.databricks_account_id + aws_vpc_endpoint_id = var.backend_rest + vpc_endpoint_name = "${var.resource_prefix}-vpce-backend-${var.vpc_id}" + region = var.region +} + +// Backend Rest VPC Endpoint Configuration +resource "databricks_mws_vpc_endpoint" "backend_relay" { + account_id = var.databricks_account_id + aws_vpc_endpoint_id = var.backend_relay + vpc_endpoint_name = "${var.resource_prefix}-vpce-relay-${var.vpc_id}" + region = var.region +} + +// Network Configuration +resource "databricks_mws_networks" "this" { + account_id = var.databricks_account_id + network_name = "${var.resource_prefix}-network" + security_group_ids = var.security_group_ids + subnet_ids = var.subnet_ids + vpc_id = var.vpc_id + vpc_endpoints { + dataplane_relay = [databricks_mws_vpc_endpoint.backend_relay.vpc_endpoint_id] + rest_api = [databricks_mws_vpc_endpoint.backend_rest.vpc_endpoint_id] + } +} + +// Managed Key Configuration +resource "databricks_mws_customer_managed_keys" "managed_storage" { + account_id = var.databricks_account_id + aws_key_info { + key_arn = var.managed_storage_key + key_alias = var.managed_storage_key_alias + } + use_cases = ["MANAGED_SERVICES"] +} + +// Workspace Storage Key Configuration +resource "databricks_mws_customer_managed_keys" "workspace_storage" { + account_id = var.databricks_account_id + aws_key_info { + key_arn = var.workspace_storage_key + key_alias = var.workspace_storage_key_alias + } + use_cases = ["STORAGE"] +} + +// Private Access Setting Configuration +resource "databricks_mws_private_access_settings" "pas" { + private_access_settings_name = "${var.resource_prefix}-PAS" + region = var.region + public_access_enabled = true + private_access_level = "ACCOUNT" +} + +// Workspace Configuration +resource "databricks_mws_workspaces" "this" { + account_id = var.databricks_account_id + aws_region = var.region + workspace_name = var.resource_prefix + # deployment_name = "development-company-A" // Deployment name for the workspace URL. This is not enabled by default on an account. Please reach out to your Databricks representative for more information. + credentials_id = databricks_mws_credentials.this.credentials_id + storage_configuration_id = databricks_mws_storage_configurations.this.storage_configuration_id + network_id = databricks_mws_networks.this.network_id + private_access_settings_id = databricks_mws_private_access_settings.pas.private_access_settings_id + managed_services_customer_managed_key_id = databricks_mws_customer_managed_keys.managed_storage.customer_managed_key_id + storage_customer_managed_key_id = databricks_mws_customer_managed_keys.workspace_storage.customer_managed_key_id + pricing_tier = "ENTERPRISE" + depends_on = [databricks_mws_networks.this] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace.tf b/aws-gov/tf/modules/sra/databricks_workspace.tf new file mode 100644 index 0000000..bc22223 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace.tf @@ -0,0 +1,116 @@ +// EXPLANATION: All modules that reside at the workspace level + +// Creates a Workspace Isolated Catalog +module "uc_catalog" { + source = "./databricks_workspace/workspace_security_modules/uc_catalog" + providers = { + databricks = databricks.created_workspace + } + + databricks_account_id = var.databricks_account_id + aws_account_id = var.aws_account_id + resource_prefix = var.resource_prefix + uc_catalog_name = "${var.resource_prefix}-catalog-${module.databricks_mws_workspace.workspace_id}" + cmk_admin_arn = var.cmk_admin_arn == null ? "arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root" : var.cmk_admin_arn + workspace_id = module.databricks_mws_workspace.workspace_id + user_workspace_catalog_admin = var.user_workspace_catalog_admin + databricks_gov_shard = var.databricks_gov_shard + databricks_prod_aws_account_id = var.databricks_prod_aws_account_id + uc_master_role_id = var.uc_master_role_id + + depends_on = [module.databricks_mws_workspace, module.uc_assignment] +} + +// Create Read-Only Storage Location for Data Bucket & External Location +module "uc_external_location" { + count = var.enable_read_only_external_location_boolean ? 1 : 0 + source = "./databricks_workspace/workspace_security_modules/uc_external_location" + providers = { + databricks = databricks.created_workspace + } + + databricks_account_id = var.databricks_account_id + aws_account_id = var.aws_account_id + resource_prefix = var.resource_prefix + read_only_data_bucket = var.read_only_data_bucket + read_only_external_location_admin = var.read_only_external_location_admin + databricks_gov_shard = var.databricks_gov_shard + databricks_prod_aws_account_id = var.databricks_prod_aws_account_id + uc_master_role_id = var.uc_master_role_id +} + +// Workspace Admin Configuration +module "admin_configuration" { + count = var.enable_admin_configs_boolean ? 1 : 0 + source = "./databricks_workspace/workspace_security_modules/admin_configuration" + providers = { + databricks = databricks.created_workspace + } +} + +// IP Access Lists - Optional +module "ip_access_list" { + source = "./databricks_workspace/workspace_security_modules/ip_access_list" + count = var.enable_ip_boolean ? 1 : 0 + providers = { + databricks = databricks.created_workspace + } + + ip_addresses = var.ip_addresses +} + +// Create Create Cluster - Optional +module "cluster_configuration" { + source = "./databricks_workspace/workspace_security_modules/cluster_configuration" + count = var.enable_cluster_boolean ? 1 : 0 + providers = { + databricks = databricks.created_workspace + } + + compliance_security_profile_egress_ports = var.compliance_security_profile_egress_ports + resource_prefix = var.resource_prefix + operation_mode = var.operation_mode +} + +// System Table Schemas Enablement - Optional +module "system_table" { + source = "./databricks_workspace/workspace_security_modules/system_schema/" + count = var.enable_system_tables_schema_boolean ? 1 : 0 + providers = { + databricks = databricks.created_workspace + } + depends_on = [ module.uc_assignment ] +} + +// SAT Implementation - Optional +module "security_analysis_tool" { + source = "./databricks_workspace/solution_accelerators/security_analysis_tool/aws" + count = var.enable_sat_boolean ? 1 : 0 + providers = { + databricks = databricks.created_workspace + } + + databricks_url = module.databricks_mws_workspace.workspace_url + workspace_id = module.databricks_mws_workspace.workspace_id + account_console_id = var.databricks_account_id + client_id = var.client_id + client_secret = var.client_secret + use_sp_auth = true + proxies = {} + analysis_schema_name = "SAT" + + depends_on = [ + module.databricks_mws_workspace + ] +} + +// System Tables Schemas - Optional +module "audit_log_alerting" { + source = "./databricks_workspace/solution_accelerators/system_tables_audit_log/" + count = var.enable_audit_log_alerting ? 1 : 0 + providers = { + databricks = databricks.created_workspace + } + + alert_emails = [var.user_workspace_admin] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf new file mode 100644 index 0000000..b055acf --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf @@ -0,0 +1,16 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} + +module "common" { + source = "../common/" + account_console_id = var.account_console_id + workspace_id = var.workspace_id + sqlw_id = var.sqlw_id + analysis_schema_name = var.analysis_schema_name + proxies = var.proxies +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf new file mode 100644 index 0000000..db695c4 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf @@ -0,0 +1,31 @@ +### AWS Specific Secrets + +resource "databricks_secret" "user" { + key = "user" + string_value = var.account_user + scope = module.common.secret_scope_id +} + +resource "databricks_secret" "pass" { + key = "pass" + string_value = var.account_pass + scope = module.common.secret_scope_id +} + +resource "databricks_secret" "use_sp_auth" { + key = "use-sp-auth" + string_value = var.use_sp_auth + scope = module.common.secret_scope_id +} + +resource "databricks_secret" "client_id" { + key = "client-id" + string_value = var.client_id + scope = module.common.secret_scope_id +} + +resource "databricks_secret" "client_secret" { + key = "client-secret" + string_value = var.client_secret + scope = module.common.secret_scope_id +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf new file mode 100644 index 0000000..bb190f7 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf @@ -0,0 +1,66 @@ +variable "databricks_url" { + description = "Should look like https://.cloud.databricks.com" + type = string +} + +variable "workspace_id" { + description = "Should be the string of numbers in the workspace URL arg (e.g. https://.cloud.databricks.com/?o=1234567890123456)" + type = string +} + +variable "account_console_id" { + description = "Databricks Account Console ID" + type = string +} + +variable "sqlw_id" { + type = string + description = "16 character SQL Warehouse ID: Type new to have one created or enter an existing SQL Warehouse ID" + validation { + condition = can(regex("^(new|[a-f0-9]{16})$", var.sqlw_id)) + error_message = "Format 16 characters (0-9 and a-f). For more details reference: https://docs.databricks.com/administration-guide/account-api/iam-role.html." + } + default = "new" +} + +### AWS Specific Variables + +variable "account_user" { + description = "Account Console Username" + type = string + default = " " +} + +variable "account_pass" { + description = "Account Console Password" + type = string + default = " " +} + +variable "use_sp_auth" { + description = "Authenticate with Service Principal OAuth tokens instead of user and password" + type = bool + default = true +} + +variable "client_id" { + description = "Service Principal Application (client) ID" + type = string + default = "value" +} + +variable "client_secret" { + description = "SP Secret" + type = string + default = "value" +} + +variable "analysis_schema_name" { + type = string + description = "Name of the schema to be used for analysis" +} + +variable "proxies" { + type = map(any) + description = "Proxies to be used for Databricks API calls" +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/data.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/data.tf new file mode 100644 index 0000000..6cdd79d --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/data.tf @@ -0,0 +1,13 @@ +data "databricks_current_user" "me" {} + +data "databricks_node_type" "smallest" { + local_disk = true + min_cores = 4 + gb_per_core = 8 + photon_worker_capable = true + photon_driver_capable = true +} + +data "databricks_spark_version" "latest_lts" { + long_term_support = true +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf new file mode 100644 index 0000000..4fdfe04 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf @@ -0,0 +1,71 @@ +resource "databricks_job" "initializer" { + name = "SAT Initializer Notebook (one-time)" + job_cluster { + job_cluster_key = "job_cluster" + new_cluster { + num_workers = 5 + spark_version = data.databricks_spark_version.latest_lts.id + node_type_id = data.databricks_node_type.smallest.id + runtime_engine = "PHOTON" + dynamic "gcp_attributes" { + for_each = var.gcp_impersonate_service_account == "" ? [] : [var.gcp_impersonate_service_account] + content { + google_service_account = var.gcp_impersonate_service_account + } + } + } + } + + task { + task_key = "Initializer" + job_cluster_key = "job_cluster" + library { + pypi { + package = "dbl-sat-sdk" + } + } + notebook_task { + notebook_path = "${databricks_repo.security_analysis_tool.workspace_path}/notebooks/security_analysis_initializer" + } + } +} + +resource "databricks_job" "driver" { + name = "SAT Driver Notebook" + job_cluster { + job_cluster_key = "job_cluster" + new_cluster { + num_workers = 5 + spark_version = data.databricks_spark_version.latest_lts.id + node_type_id = data.databricks_node_type.smallest.id + runtime_engine = "PHOTON" + dynamic "gcp_attributes" { + for_each = var.gcp_impersonate_service_account == "" ? [] : [var.gcp_impersonate_service_account] + content { + google_service_account = var.gcp_impersonate_service_account + } + } + } + } + + + task { + task_key = "Driver" + job_cluster_key = "job_cluster" + library { + pypi { + package = "dbl-sat-sdk" + } + } + notebook_task { + notebook_path = "${databricks_repo.security_analysis_tool.workspace_path}/notebooks/security_analysis_driver" + } + } + + schedule { + #E.G. At 08:00:00am, on every Monday, Wednesday and Friday, every month; For more: http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html + quartz_cron_expression = "0 0 8 ? * Mon,Wed,Fri" + # The system default is UTC; For more: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones + timezone_id = "America/New_York" + } +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/outputs.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/outputs.tf new file mode 100644 index 0000000..11dda97 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/outputs.tf @@ -0,0 +1,4 @@ +output "secret_scope_id" { + value = databricks_secret_scope.sat.id + description = "ID of the created secret scope to add more secrets if necessary" +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf new file mode 100644 index 0000000..1d847d2 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf new file mode 100644 index 0000000..7b21149 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf @@ -0,0 +1,7 @@ +#Make sure Files in Repos option is enabled in Workspace Admin Console > Workspace Settings + +resource "databricks_repo" "security_analysis_tool" { + url = "https://github.com/databricks-industry-solutions/security-analysis-tool.git" + branch = "main" + path = "/Workspace/Applications/SAT_TF" +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf new file mode 100644 index 0000000..c905f1c --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf @@ -0,0 +1,33 @@ +resource "databricks_secret_scope" "sat" { + name = var.secret_scope_name +} + +resource "databricks_secret" "user_email" { + key = "user-email-for-alerts" + string_value = var.notification_email == "" ? data.databricks_current_user.me.user_name : var.notification_email + scope = databricks_secret_scope.sat.id +} + +resource "databricks_secret" "account_console_id" { + key = "account-console-id" + string_value = var.account_console_id + scope = databricks_secret_scope.sat.id +} + +resource "databricks_secret" "sql_warehouse_id" { + key = "sql-warehouse-id" + string_value = var.sqlw_id == "new" ? databricks_sql_endpoint.new[0].id : data.databricks_sql_warehouse.old[0].id + scope = databricks_secret_scope.sat.id +} + +resource "databricks_secret" "analysis_schema_name" { + key = "analysis_schema_name" + string_value = var.analysis_schema_name + scope = databricks_secret_scope.sat.id +} + +resource "databricks_secret" "proxies" { + key = "proxies" + string_value = jsonencode(var.proxies) + scope = databricks_secret_scope.sat.id +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/sql_warehouse.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/sql_warehouse.tf new file mode 100644 index 0000000..91b4dae --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/sql_warehouse.tf @@ -0,0 +1,18 @@ +resource "databricks_sql_endpoint" "new" { + count = var.sqlw_id == "new" ? 1 : 0 + name = "SAT Warehouse" + cluster_size = "Small" + max_num_clusters = 1 + + tags { + custom_tags { + key = "owner" + value = data.databricks_current_user.me.alphanumeric + } + } +} + +data "databricks_sql_warehouse" "old" { + count = var.sqlw_id == "new" ? 0 : 1 + id = var.sqlw_id +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf new file mode 100644 index 0000000..150ac5a --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf @@ -0,0 +1,46 @@ +variable "account_console_id" { + type = string + description = "Databricks Account ID" +} + +variable "workspace_id" { + description = "Should be the string of numbers in the workspace URL arg (e.g. https://.azuredatabricks.net/?o=1234567890123456)" +} + +variable "sqlw_id" { + type = string + description = "16 character SQL Warehouse ID: Type new to have one created or enter an existing SQL Warehouse ID" + validation { + condition = can(regex("^(new|[a-f0-9]{16})$", var.sqlw_id)) + error_message = "Format 16 characters (0-9 and a-f). For more details reference: https://docs.databricks.com/administration-guide/account-api/iam-role.html." + } + default = "new" +} + +variable "secret_scope_name" { + description = "Name of secret scope for SAT secrets" + type = string + default = "sat_scope" +} + +variable "notification_email" { + type = string + description = "Optional user email for notifications. If not specified, current user's email will be used" + default = "" +} + +variable "gcp_impersonate_service_account" { + type = string + description = "GCP Service Account to impersonate (e.g. xyz-sa-2@project.iam.gserviceaccount.com)" + default = "" +} + +variable "analysis_schema_name" { + type = string + description = "Name of the schema to be used for analysis" +} + +variable "proxies" { + type = map(any) + description = "Proxies to be used for Databricks API calls" +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/job.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/job.tf new file mode 100644 index 0000000..7dee1cf --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/job.tf @@ -0,0 +1,34 @@ +resource "databricks_job" "this" { + name = "System Tables" + + dynamic "task" { + for_each = local.alerts + content { + task_key = task.value + + sql_task { + warehouse_id = local.warehouse_id + alert { + alert_id = databricks_sql_alert.alert[task.value].id + + dynamic "subscriptions" { + for_each = var.alert_emails + content { + user_name = subscriptions.value + } + } + } + } + } + } + + schedule { + quartz_cron_expression = "1 1 * * * ?" + timezone_id = "UTC" + } + + tags = { + project = "system-tables" + owner = data.databricks_current_user.me.user_name + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/main.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/main.tf new file mode 100644 index 0000000..182aa32 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/main.tf @@ -0,0 +1,14 @@ +data "databricks_current_user" "me" {} + +locals { + qa_data = jsondecode(file("${path.module}/queries_and_alerts.json"))["queries_and_alerts"] + directories = toset(compact(flatten([for k in local.qa_data : [k.parent, try(k.alert.parent, null)]]))) + queries = toset([for k in local.qa_data : k.name]) + alerts = toset([for k in local.qa_data : k.name if try(k.alert, null) != null]) + data_map = { for k in local.qa_data : k.name => k } +} + +resource "databricks_directory" "this" { + for_each = local.directories + path = "${data.databricks_current_user.me.home}/${each.value}" +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/provider.tf new file mode 100644 index 0000000..1d847d2 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/queries_and_alerts.json b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/queries_and_alerts.json new file mode 100644 index 0000000..eec87a3 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/queries_and_alerts.json @@ -0,0 +1,664 @@ +{ + "queries_and_alerts": [ + { + "name": "repeated_failed_login_attempts", + "description": "Repeated failed login attempts could indicate an attacker trying to brute force access to your lakehouse. The following query can be used to detect repeated failed login attempts over a 60 minute period within the last 24 hours.", + "query": "SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, ifnull(user_identity.email, request_params.user) AS email, collect_set(action_name) AS action_names, collect_set(response.error_message) AS error_messages, collect_set(response.status_code) AS response_codes, count(*) AS total FROM system.access.audit WHERE action_name IN ('aadBrowserLogin', 'aadTokenLogin', 'certLogin', 'jwtLogin', 'login', 'oidcBrowserLogin', 'samlLogin', 'tokenLogin') AND response.status_code IN (401, 403) AND WINDOW(event_time, '60 minutes').end >= current_timestamp() - INTERVAL 24 HOURS GROUP BY 1, 2, 3 ORDER BY total DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "repeated_failed_login_attempts", + "options": { + "column": "total", + "custom_body": "

There have been the following failed login attempts within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "1" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "failed_login_attempts_last_90_days", + "description": "Repeated failed login attempts could indicate an attacker trying to brute force access to your lakehouse.", + "query": "SELECT event_date, ifnull(user_identity.email, request_params.user) AS email, workspace_id, action_name, count(*) AS num_failed_logins FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND action_name IN ('aadBrowserLogin', 'aadTokenLogin', 'certLogin', 'jwtLogin', 'login', 'oidcBrowserLogin', 'samlLogin', 'tokenLogin') AND response.status_code IN (401, 403) GROUP BY 1, 2, 3, 4 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "changes_to_admin_users", + "description": "Databricks account and workspace admins should be limited to a few very trusted individuals responsible for managing the deployment. The granting of new admin privileges should be reviewed. The following query can be used to detect changes to admin users within the last 24 hours.", + "query": "SELECT event_time, workspace_id, user_identity.email, lower(replace(audit_level, '_LEVEL', '')) AS account_or_workspace, action_name, request_params.targetUserName, request_params.targetGroupName, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND (action_name IN ('setAccountAdmin', 'changeAccountOwner', 'setAdmin', 'removeAdmin') OR (action_name IN ('addPrincipalToGroup', 'removePrincipalFromGroup') AND request_params.targetGroupName = 'admins')) GROUP BY 1, 2, 3, 4, 5, 6, 7 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "changes_to_admin_users", + "options": { + "column": "total", + "custom_body": "

There have been the following changes to admin users within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "changes_to_workspace_configuration", + "description": "Many workspace-level configurations perform a security-enforcing function. The following SQL query can be used to detect changes in workspace configuration within the last 24 hours.", + "query": "SELECT event_time, user_identity.email, workspace_id, request_params.workspaceConfKeys, request_params.workspaceConfValues, count(*) AS total FROM system.access.audit WHERE action_name = 'workspaceConfEdit' AND event_time >= current_timestamp() - INTERVAL 24 HOURS GROUP BY 1, 2, 3, 4, 5 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "changes_to_workspace_configuration", + "options": { + "column": "total", + "custom_body": "

There have been the following changes to workspace configurations within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "data_downloads_from_control_plane", + "description": "Databricks allows customers to configure whether they want users to be able to download notebook or SQL query results, but some customers might want to monitor and report rather than prevent entirely. The following query can be used to detect high numbers of downloads of results from notebooks, Databricks SQL, Unity Catalog volumes and MLflow, as well as the exporting of notebooks in formats that may contain query results within the last 24 hours.", + "query": "with downloads AS (SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, collect_set(workspace_id) AS workspace_ids, collect_set(service_name) AS service_names, collect_set(action_name) AS action_names, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND (action_name IN ('downloadPreviewResults', 'downloadLargeResults', 'filesGet', 'getModelVersionDownloadUri', 'getModelVersionSignedDownloadUri') OR (action_name = 'workspaceExport' AND request_params.workspaceExportFormat != 'SOURCE') OR (action_name = 'downloadQueryResult' AND request_params.fileType != 'arrows')) GROUP BY 1, 2, 3) SELECT * FROM downloads WHERE total > 20 ORDER BY total DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "data_downloads_from_control_plane", + "options": { + "column": "total", + "custom_body": "

There have been the following high number of downloads from the control plane within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "20" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "data_downloads_from_control_plane_last_90_days", + "description": "Spikes in the number of downloads could indicate attempts to exfiltrate data.", + "query": "SELECT event_date, ifnull(user_identity.email, request_params.user) AS email, workspace_id, action_name, count(*) AS number_of_downloads FROM system.access.audit WHERE event_time >= current_date() - INTERVAL 90 DAYS AND (action_name IN ('downloadPreviewResults', 'downloadLargeResults', 'filesGet', 'getModelVersionDownloadUri', 'getModelVersionSignedDownloadUri') OR (action_name = 'workspaceExport' AND request_params.workspaceExportFormat != 'SOURCE') OR (action_name = 'downloadQueryResult' AND request_params.fileType != 'arrows')) GROUP BY 1, 2, 3, 4 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "ip_access_list_failures", + "description": "Databricks allows customers to configure IP Access Lists to restrict access to their account & workspaces. However, they may want to monitor and be alerted whenever access is attempted from an untrusted network. The following query can be used to detect all IpAccessDenied and accountIpAclsValidationFailed events within the last 24 hours.", + "query": "SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, workspace_id, ifnull(user_identity.email, request_params.user) AS email, source_ip_address, collect_set(action_name) AS action_names, collect_set(response.error_message) AS error_messages, collect_set(request_params.path) AS urls, collect_set(response.status_code) AS status_codes, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name IN ('IpAccessDenied', 'accountIpAclsValidationFailed') GROUP BY 1, 2, 3, 4, 5 ORDER BY total DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "ip_access_list_failures", + "options": { + "column": "total", + "custom_body": "

There have been the following attempts to access the control plane from unauthorized networks within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "ip_access_list_failures_last_90_days", + "description": "Repeated IP access list failures could indicate attempts to brute force access to your lakehouse, or internal users trying to connect from untrusted networks.", + "query": "SELECT event_date, workspace_id, source_ip_address, count(*) AS number_of_failures FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND action_name IN ('IpAccessDenied', 'accountIpAclsValidationFailed') GROUP BY 1, 2, 3 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "ip_access_list_changes", + "description": "Databricks allows customers to configure IP access lists to restrict access to their account & workspaces. However, they may want to monitor and be alerted whenever thos IP access lists change. The following query can be used to detect all createIpAccessList, deleteIpAccessList and updateIpAccessList events within the last 24 hours.", + "query": "SELECT event_time, user_identity.email, workspace_id, action_name, request_params.ipAccessListId, response.status_code, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name IN ('createIpAccessList', 'deleteIpAccessList', 'updateIpAccessList') GROUP BY 1, 2, 3, 4, 5, 6 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "ip_access_list_changes", + "options": { + "column": "total", + "custom_body": "

There have been the following attempts to change the IP access list settings of a workspace within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "databricks_access_to_customer_workspaces", + "description": "This query can be used to detect logins to your workspace via the Databricks support process. This access is tied to a support ticket while also complying with your workspace configuration that may disable such access. The following query can be used to detect Databricks access to your workspaces within the last 24 hours.", + "query": "SELECT event_time, workspace_id, request_params.user, request_params.approver, request_params.duration, request_params.reason, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name = 'databricksAccess' GROUP BY 1, 2, 3, 4, 5, 6 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "databricks_access_to_customer_workspaces", + "options": { + "column": "total", + "custom_body": "

There have been the following logins to your workspaces from Databricks employees within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "databricks_access_to_customer_workspaces_last_90_days", + "description": "All logins to your workspace via the Databricks support process. This access is tied to a support ticket while also complying with your workspace configuration that may disable such access.", + "query": "SELECT event_time, workspace_id, request_params.user, request_params.approver, request_params.duration, request_params.reason, count(*) AS total FROM system.access.audit WHERE event_time >= current_date() - INTERVAL 90 DAYS AND action_name = 'databricksAccess' GROUP BY 1, 2, 3, 4, 5, 6 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "terms_of_service_changes", + "description": "As Databricks rolls out new products and features, customers may occassionally have to agree to changes in our Terms of Service before they can opt-in to the new feature. Some customers might want to monitor when an account admin accepts such terms of service changes. The following SQL query can be used to detect any acceptance or sending of Terms of Service changes within the last 24 hours", + "query": "SELECT event_time, user_identity.email, request_params.account_id, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, action_name, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name IN ('acceptTos', 'sendTos') GROUP BY 1, 2, 3, 4, 5 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "terms_of_service_changes", + "options": { + "column": "total", + "custom_body": "

There have been the following Terms of Service changes detected within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "account_settings_changes", + "description": "Many account-level settings perform a security-enforcing function. The following SQL query can be used to detect changes in account level settings within the last 24 hours.", + "query": "SELECT event_time, user_identity.email, account_id, request_params.settingTypeName AS setting_name, request_params.settingValueForAudit AS setting_value, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND audit_level = 'ACCOUNT_LEVEL' AND service_name = 'accounts' AND action_name = 'setSetting' GROUP BY 1, 2, 3, 4, 5 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "account_settings_changes", + "options": { + "column": "total", + "custom_body": "

There have been the following account settings changes detected within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "global_init_script_changes", + "description": "Global init scripts run arbitrary code that is executed on every cluster. This can be a very powerful capability but with great power comes great responsibility. The following SQL query can be used to detect the creation, update and deletion of global init scripts within the last 24 hours.", + "query": "SELECT event_time, workspace_id, user_identity.email, source_ip_address, action_name, request_params.name, request_params.script_id, request_params.enabled, request_params.`script-SHA256`, response.status_code, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'globalInitScripts' GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "global_init_script_changes", + "options": { + "column": "total", + "custom_body": "

There have been the following changes to global init scripts within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "install_library_on_all_clusters", + "description": "Installing libraries on all clusters is an anti-pattern. Customers should use cluster-scoped or notebook-scoped libraries for many different reasons including but not limited to transparency, recreatability, reliability and security. The following SQL query can be used to detect any attempts to install libraries on all clusters within the last 24 hours.", + "query": "SELECT event_time, workspace_id, user_identity.email, request_params.library, response.status_code, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'clusterLibraries' AND action_name = 'installLibraryOnAllClusters' GROUP BY 1, 2, 3, 4, 5 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "install_library_on_all_clusters", + "options": { + "column": "total", + "custom_body": "

There have been the following attempts to install libraries on all clusters detected within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "mount_point_creation", + "description": "Mount points are considered an anti-pattern because mount points do not have the same strong data governance features as external locations or volumes in Unity Catalog. The following query can be used to detect new mount points created or changed within the last 24 hours", + "query": "SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, collect_set(workspace_id) AS workspace_ids, collect_set(request_params.mountPoint) AS mount_points, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name = 'mount' GROUP BY 1, 2, 3 ORDER BY total DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "mount_point_creation", + "options": { + "column": "total", + "custom_body": "

There have been the following mount points created within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "long_lifetime_token_generation", + "description": "Personal access tokens should be treated like a credential and protected at all times. As well as being managed by the Token Management API and secured with additional protections like IP Access Lists, they should only be generated with a short lifetime. The following SQL query can be used to detect the generation of PAT tokens with a lifetime of greater than 72 hours.", + "query": "SELECT event_time, workspace_id, user_identity.email, timestamp_millis(cast(request_params.tokenExpirationTime AS BIGINT)) AS token_expiration, timestampdiff(HOUR, event_time, timestamp_millis(cast(request_params.tokenExpirationTime AS BIGINT))) AS token_duration_in_hours, request_params.tokenHash, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name = 'generateDbToken' AND timestampdiff(HOUR, event_time, timestamp_millis(cast(request_params.tokenExpirationTime AS BIGINT))) > 72 GROUP BY 1, 2, 3, 4, 5, 6 ORDER BY event_time DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "long_lifetime_token_generation", + "options": { + "column": "total", + "custom_body": "

There have been the following tokens generated with a lifetime of >72 hours within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "destructive_activities", + "description": "A high number of destructive activities (such as delete* events) may indicate a malicious attempt to cause disruption and harm. The following SQL query can be used to detect users who have attempted a high number (>50) destructive activities within the last 24 hours. This query filters out activities from Databricks System-Users, although you could optionally add them back in.", + "query": "SELECT * FROM (SELECT event_date, user_identity.email, collect_set(if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id)) AS workspace_ids, collect_set(service_name) AS service_names, size(collect_set(service_name)) AS num_services, collect_set(action_name) AS action_names, size(collect_set(action_name)) AS num_actions, count(*) AS num_destructive_activities FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND user_identity.email NOT IN ('System-User') AND (startswith(action_name, 'delete') OR contains(lower(action_name), 'delete') OR contains(lower(action_name), 'trash')) GROUP BY 1, 2) WHERE num_destructive_activities > 50 ORDER BY num_destructive_activities DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "destructive_activities", + "options": { + "column": "num_destructive_activities", + "custom_body": "

There have been the following high numbers of destructive activities detected within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">=", + "value": "50" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "destructive_activities_last_90_days", + "description": "A spike in the number of destructive activities (such as delete* events) may indicate a malicious attempt to cause disruption and harm.", + "query": "SELECT event_date, user_identity.email, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, service_name, action_name, count(*) AS num_destructive_activities FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND user_identity.email NOT IN ('System-User') AND (startswith(action_name, 'delete') OR contains(lower(action_name), 'delete') OR contains(lower(action_name), 'trash')) GROUP BY 1, 2, 3, 4, 5 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "potential_privilege_escalation", + "description": "A high number of permission changes could indicate privelege escalation. The following SQL query can be used to detect users who have made a high number (>25) within an hour period over the last 24 hours. This query filters out changes made by Databricks System-Users, although you could optionally add them back in.", + "query": "SELECT * FROM (SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, collect_set(if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id)) AS workspace_ids, collect_set(service_name) AS service_names, size(collect_set(service_name)) AS num_services, collect_set(action_name) AS action_names, size(collect_set(action_name)) AS num_actions, count(*) AS num_permissions_changes FROM system.access.audit WHERE action_name IN ('addPrincipalToGroup', 'changeDatabricksSqlAcl', 'changeDatabricksWorkspaceAcl', 'changeDbTokenAcl', 'changePasswordAcl', 'changeServicePrincipalAcls', 'generateDbToken', 'setAdmin', 'changeClusterAcl', 'changeClusterPolicyAcl', 'changeWarehouseAcls', 'changePermissions', 'transferObjectOwnership', 'changePipelineAcls', 'changeFeatureTableAcl', 'addPrincipalToGroup', 'changeIamRoleAcl', 'changeInstancePoolAcl', 'changeJobAcl', 'resetJobAcl', 'changeRegisteredModelAcl', 'changeInferenceEndpointAcl', 'putAcl', 'changeSecurableOwner', 'grantPermission', 'changeWorkspaceAcl', 'updateRoleAssignment', 'setAccountAdmin', 'changeAccountOwner', 'updatePermissions', 'updateSharePermissions') AND event_time >= current_timestamp() - INTERVAL 24 HOURS AND user_identity.email NOT IN ('System-User') GROUP BY 1, 2, 3) WHERE num_permissions_changes > 25 ORDER BY num_permissions_changes DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "potential_privilege_escalation", + "options": { + "column": "num_permissions_changes", + "custom_body": "

There have been the following high numbers of permissions changes detected within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">=", + "value": "25" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "potential_privilege_escalation_last_90_days", + "description": "A spike in the number of permission changes could indicate privilege escalation.", + "query": "SELECT event_date, user_identity.email, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, service_name, action_name, count(*) AS num_permissions_changes FROM system.access.audit WHERE action_name IN ('addPrincipalToGroup', 'changeDatabricksSqlAcl', 'changeDatabricksWorkspaceAcl', 'changeDbTokenAcl', 'changePasswordAcl', 'changeServicePrincipalAcls', 'generateDbToken', 'setAdmin', 'changeClusterAcl', 'changeClusterPolicyAcl', 'changeWarehouseAcls', 'changePermissions', 'transferObjectOwnership', 'changePipelineAcls', 'changeFeatureTableAcl', 'addPrincipalToGroup', 'changeIamRoleAcl', 'changeInstancePoolAcl', 'changeJobAcl', 'resetJobAcl', 'changeRegisteredModelAcl', 'changeInferenceEndpointAcl', 'putAcl', 'changeSecurableOwner', 'grantPermission', 'changeWorkspaceAcl', 'updateRoleAssignment', 'setAccountAdmin', 'changeAccountOwner', 'updatePermissions', 'updateSharePermissions') AND event_date >= current_date() - INTERVAL 90 DAYS AND user_identity.email NOT IN ('System-User') GROUP BY 1, 2, 3, 4, 5 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "repeated_access_to_secrets", + "description": "Repeated attempts to access secrets could indicate an attempt to steal credentials. The following SQL query can be used to detect users who have attempted a high number (>10) of attempts to access secrets within an hour period over the last 24 hours. This query filters out requests from Databricks System-Users, although you could optionally add them back in.", + "query": "SELECT * FROM (SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, collect_set(workspace_id) AS workspace_ids, size(collect_set(request_params.scope)) AS num_scopes_accessed, collect_set(request_params.scope) AS secret_scopes, size(collect_set(request_params.key)) AS num_keys_accessed, collect_set(request_params.key) AS secret_keys, count(*) AS num_requests FROM system.access.audit WHERE action_name = 'getSecret' AND event_time >= current_timestamp() - INTERVAL 24 HOURS AND user_identity.email NOT IN ('System-User') GROUP BY 1, 2, 3) WHERE num_keys_accessed >= 10 ORDER BY num_keys_accessed DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "repeated_access_to_secrets", + "options": { + "column": "num_keys_accessed", + "custom_body": "

There have been the repeated attempts to access secrets within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">=", + "value": "10" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "access_to_secrets_last_90_days", + "description": "A spike in the number of requests to access secrets could indicate attempts to steal credentials.", + "query": "SELECT event_date, user_identity.email, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, concat(request_params.scope, '/', request_params.key) AS secret, count(*) AS num_requests FROM system.access.audit WHERE action_name = 'getSecret' AND event_date >= current_date() - INTERVAL 90 DAYS AND user_identity.email NOT IN ('System-User') GROUP BY 1, 2, 3, 4 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "access_to_multiple_workspaces", + "description": "The same user accessing multiple workspaces within a short time frame could indicate lateral movement, or malicious attempts to increase the blast radius of an attack. The following SQL query can be used to detect users who have accessed a high number (>5) of different workspaces within the last 24 hours. This query filters out requests from unknown and Databricks System-Users, although you could optionally add them back in.", + "query": "SELECT * FROM (SELECT event_date, user_identity.email, collect_set(workspace_id) AS workspace_ids, count(distinct workspace_id) AS num_workspaces_accessed FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND user_identity.email NOT IN ('System-User', 'unknown') GROUP BY 1, 2) WHERE num_workspaces_accessed >= 5 ORDER BY num_workspaces_accessed DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "access_to_multiple_workspaces", + "options": { + "column": "num_workspaces_accessed", + "custom_body": "

There have been the following attempts to access multiple workspaces within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">=", + "value": "5" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "use_of_print_statements", + "description": "Databricks supports verbose audit logging, which can be useful in highly regulated environments in which all commands run interactively by a user must be recorded. Verbose audit logs can also be useful for monitoring compliance with coding standards. For example, let's suppose your organization has a policy that print() statements should not be used, the following SQL query could be used to monitor compliance with such a policy by detecting uses of the print() statement within the last 24 hours.", + "query": "SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, collect_set(workspace_id) AS workspace_ids, collect_set(service_name) AS service_names, collect_set(request_params.commandLanguage) AS command_languages, collect_set(request_params.commandText) AS commands, collect_set(request_params.status) AS statuses, collect_set(request_params.notebookId) AS notebook_ids, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name = 'runCommand' AND request_params.commandText rlike 'print[/s]?(?! e)(.)+' GROUP BY 1, 2, 3 ORDER BY total DESC", + "parent": "system_tables/audit/admin/queries/", + "alert": { + "name": "use_of_print_statements", + "options": { + "column": "total", + "custom_body": "

There have been the following use of print statements within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/admin/alerts/" + } + }, + { + "name": "ip_addresses_used_to_access_databricks", + "description": "The following SQL query will show you which IP addresses and the number of requests for each have been used to access your workspace or account over the last 90 days.", + "query": "SELECT regexp_replace(source_ip_address, '(:\\\\d*)', '') AS source_ip_address, CASE WHEN audit_level = 'ACCOUNT_LEVEL' AND service_name != 'unityCatalog' THEN 'account' WHEN audit_level = 'ACCOUNT_LEVEL' AND service_name = 'unityCatalog' THEN 'unity_catalog' WHEN audit_level = 'WORKSPACE_LEVEL' THEN 'workspace' ELSE NULL END AS service, count(*) AS total_requests FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND source_ip_address NOT IN ('', '0.0.0.0', '127.0.0.1') GROUP BY 1, 2 ORDER BY total_requests DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "ip_address_ranges_used_to_access_databricks", + "description": "The following SQL query will show you which IP address ranges and the number of requests for each have been used to access your workspaces or account over the last 90 days.", + "query": "SELECT concat(substring_index(source_ip_address, '.', 3), '.0/24') AS source_ip_range, CASE WHEN audit_level = 'ACCOUNT_LEVEL' AND service_name != 'unityCatalog' THEN 'account' WHEN audit_level = 'ACCOUNT_LEVEL' AND service_name = 'unityCatalog' THEN 'unity_catalog' WHEN audit_level = 'WORKSPACE_LEVEL' THEN 'workspace' ELSE NULL END AS service, count(*) AS total_requests FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND source_ip_address NOT IN ('', '0.0.0.0', '127.0.0.1') GROUP BY concat(substring_index(source_ip_address, '.', 3), '.0/24'), 2 ORDER BY total_requests DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "repeated_unauthorized_uc_requests", + "description": "Repeated unauthorized UC requests could indicate privilege escalation, data exfiltration attempts or an attacker trying to brute force access to your data. The following query can be used to detect repeated unauthorized UC requests over a 60 minute period within the last 24 hours.", + "query": "WITH failed_requests AS (SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, request_params.metastore_id, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, action_name, response.error_message FROM system.access.audit WHERE service_name = 'unityCatalog' AND response.status_code IN (401, 403) AND WINDOW(event_time, '60 minutes').end >= current_timestamp() - INTERVAL 24 HOURS), failed_requests_agg AS (SELECT window_start, window_end, email, metastore_id, collect_set(workspace_id) AS workspace_ids,collect_set(action_name) AS action_names, collect_set(error_message) AS error_messages, count(*) AS total FROM failed_requests GROUP BY 1, 2, 3, 4) SELECT * FROM failed_requests_agg WHERE total > 25 ORDER BY total DESC", + "parent": "system_tables/audit/unity_catalog/queries/", + "alert": { + "name": "repeated_unauthorized_uc_requests", + "options": { + "column": "total", + "custom_body": "

There have been the following unauthorized UC requests within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "25" + }, + "rearm": "3600", + "parent": "system_tables/audit/unity_catalog/alerts/" + } + }, + { + "name": "repeated_unauthorized_uc_data_requests", + "description": "Repeated unauthorized UC data requests could indicate privilege escalation, data exfiltration attempts or an attacker trying to brute force access to your data. The following query can be used to detect repeated unauthorized UC data access ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') requests over a 60 minute period within the last 24 hours.", + "query": "WITH failed_data_access AS (SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end,user_identity.email, request_params.metastore_id, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, action_name, CASE WHEN isnotnull(request_params.table_full_name) THEN request_params.table_full_name WHEN isnotnull(request_params.volume_full_name) THEN request_params.volume_full_name WHEN isnotnull(request_params.name) THEN request_params.name WHEN isnotnull(request_params.url) THEN request_params.url WHEN isnotnull(request_params.table_url) THEN request_params.table_url WHEN isnotnull(request_params.table_id) THEN request_params.table_id WHEN isnotnull(request_params.volume_id) THEN request_params.volume_id ELSE NULL END AS securable, response.error_message FROM system.access.audit WHERE action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') AND response.status_code IN (401, 403) AND WINDOW(event_time, '60 minutes').end >= current_timestamp() - INTERVAL 24 HOURS), failed_data_access_agg AS (SELECT window_start, window_end, email, metastore_id, collect_set(workspace_id) AS workspace_ids, collect_set(action_name) AS action_names, collect_set(securable) AS securables, collect_set(error_message) AS errors, count(*) AS total FROM failed_data_access GROUP BY 1, 2, 3, 4) SELECT * FROM failed_data_access_agg WHERE total > 15 ORDER BY total DESC", + "parent": "system_tables/audit/unity_catalog/queries/", + "alert": { + "name": "repeated_unauthorized_uc_data_requests", + "options": { + "column": "total", + "custom_body": "

There have been the following unauthorized UC data requests within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "15" + }, + "rearm": "3600", + "parent": "system_tables/audit/unity_catalog/alerts/" + } + }, + { + "name": "unauthorized_uc_data_requests_last_90_days", + "description": "Repeated unauthorized UC data requests could indicate privilege escalation, data exfiltration attempts or an attacker trying to brute force access to your data.", + "query": "SELECT event_date, ifnull(user_identity.email, request_params.user) AS email, request_params.workspace_id, action_name, count(*) AS total FROM system.access.audit WHERE action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') AND response.status_code IN (401, 403) AND event_date >= current_date() - INTERVAL 90 DAYS GROUP BY 1, 2, 3, 4 ORDER BY event_date DESC", + "parent": "system_tables/audit/unity_catalog/queries/" + }, + { + "name": "high_number_of_read_writes", + "description": "A high number of read/writes, particularly where the writes are to different locations could indicate data exfiltration attempts. The following query can be used to detect a high number of read/writes of UC securables (>20) within an hour window over the last 24 hours, particularly where the user is writing to different locations to the reads.", + "query": "WITH read_writes AS (SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, user_identity.email, request_params.metastore_id, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, CASE WHEN contains(request_params.operation, 'CREATE') THEN 'WRITE' WHEN contains(request_params.operation, 'WRITE') THEN 'WRITE' WHEN contains(request_params.operation, 'READ') THEN 'READ' ELSE NULL END AS operation, request_params.operation AS full_operation, CASE WHEN isnotnull(request_params.table_full_name) THEN request_params.table_full_name WHEN isnotnull(request_params.volume_full_name) THEN request_params.volume_full_name WHEN isnotnull(request_params.url) THEN request_params.url WHEN isnotnull(request_params.table_url) THEN request_params.table_url WHEN isnotnull(request_params.table_id) THEN request_params.table_id WHEN isnotnull(request_params.volume_id) THEN request_params.volume_id ELSE NULL END AS securable FROM system.access.audit WHERE action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential') AND WINDOW(event_time, '60 minutes').end >= current_timestamp() - INTERVAL 24 HOURS), read_writes_agg AS (SELECT window_start, window_end, email, metastore_id, collect_set(workspace_id) AS workspace_ids,collect_set(operation) AS operations, collect_set(securable) FILTER(WHERE operation = 'READ') AS read_securables, collect_set(securable) FILTER(WHERE operation = 'WRITE') AS write_securables, count(distinct securable) FILTER(WHERE operation = 'READ') AS num_reads, count(distinct securable) FILTER(WHERE operation = 'WRITE') AS num_writes, count(distinct operation, securable) AS total_read_writes FROM read_writes GROUP BY 1, 2, 3, 4 ORDER BY total_read_writes DESC), read_writes_high AS (SELECT window_start, window_end, email, metastore_id, workspace_ids, operations, read_securables, write_securables, array_except(write_securables, read_securables) AS writes_to_different_locations, num_reads, num_writes, size(array_except(write_securables, read_securables)) AS num_writes_to_different_locations, total_read_writes FROM read_writes_agg) SELECT * FROM read_writes_high WHERE num_reads > 20 AND num_writes > 20 AND num_writes_to_different_locations > 0 ORDER BY num_writes_to_different_locations DESC", + "parent": "system_tables/audit/unity_catalog/queries/", + "alert": { + "name": "high_number_of_read_writes", + "options": { + "column": "num_writes_to_different_locations", + "custom_body": "

There have been the following high number of reads/writes to UC securables within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/unity_catalog/alerts/" + } + }, + { + "name": "read_writes_last_90_days", + "description": "A spike in the number of read/writes (particularly writes) could indicate attempts to exfiltrate data.", + "query": "SELECT event_date, user_identity.email, if(isnotnull(request_params.workspace_id), request_params.workspace_id, workspace_id) AS workspace_id, CASE WHEN contains(request_params.operation, 'CREATE') THEN 'WRITE' WHEN contains(request_params.operation, 'WRITE') THEN 'WRITE' WHEN contains(request_params.operation, 'READ') THEN 'READ' ELSE NULL END AS operation, request_params.operation AS full_operation, CASE WHEN isnotnull(request_params.table_full_name) THEN request_params.table_full_name WHEN isnotnull(request_params.volume_full_name) THEN request_params.volume_full_name WHEN isnotnull(request_params.url) THEN request_params.url WHEN isnotnull(request_params.table_url) THEN request_params.table_url WHEN isnotnull(request_params.table_id) THEN request_params.table_id WHEN isnotnull(request_params.volume_id) THEN request_params.volume_id ELSE NULL END AS securable, COUNT(*) AS number_of_read_writes FROM system.access.audit WHERE action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential') AND event_date >= current_date() - INTERVAL 90 DAYS GROUP BY 1, 2, 3, 4, 5, 6 ORDER BY event_date DESC", + "parent": "system_tables/audit/admin/queries/" + }, + { + "name": "delta_sharing_recipients_without_ip_acls", + "description": "If you’re sharing personal data, delta sharing recipients should always be secured with IP access lists. The following SQL query can be used to detect the creation or update of delta sharing recipients which do not have IP access lists defined within the last 24 hours.", + "query": "SELECT event_time, user_identity.email, CASE WHEN request_params.name IS NOT NULL THEN request_params.name WHEN request_params.name_arg IS NOT NULL THEN request_params.name_arg ELSE NULL END AS delta_share, request_params.ip_access_list, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name IN ('createRecipient') AND request_params.ip_access_list IS NULL GROUP BY 1, 2, 3, 4 ORDER BY event_time DESC", + "parent": "system_tables/audit/unity_catalog/queries/", + "alert": { + "name": "delta_sharing_recipients_without_ip_acls", + "options": { + "column": "total", + "custom_body": "

There have been the following Delta Sharing recipients created without IP ACLs within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/unity_catalog/alerts/" + } + }, + { + "name": "delta_sharing_ip_access_list_failures", + "description": "If you’re sharing personal data, delta sharing recipients should always be secured with IP access lists. The following SQL query can be used to detect Delta Sharing data access requests ('deltaSharingQueryTable', 'deltaSharingQueryTableChanges') which have failed IP access list checks within the last 24 hours.", + "query": "SELECT WINDOW(event_time, '60 minutes').start AS window_start, WINDOW(event_time, '60 minutes').end AS window_end, source_ip_address, request_params.metastore_id, collect_set(request_params.name) AS share_names, collect_set(request_params.share) AS shares, collect_set(request_params.recipient_name) AS recipient_names, collect_set(request_params.recipient_authentication_type) AS authentication_types, COUNT(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'unityCatalog' AND action_name IN ('deltaSharingQueryTable', 'deltaSharingQueryTableChanges') AND request_params.is_ip_access_denied = 'true' GROUP BY 1, 2, 3, 4 ORDER BY total DESC", + "parent": "system_tables/audit/unity_catalog/queries/", + "alert": { + "name": "delta_sharing_ip_access_list_failures", + "options": { + "column": "total", + "custom_body": "

There have been the following Delta Sharing data access requests which have failed IP access list rules within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/unity_catalog/alerts/" + } + }, + { + "name": "delta_sharing_recipient_token_lifetime_change", + "description": "Delta Sharing recipient tokens are valid for the lifetime that you specify. As well as protecting Delta Shares via IP access lists, you should also ensure that the lifetime of a recipient token is set to a value that is suitable for the data within the metastore it is accessing. Once you have set a token lifetime, you may want to monitor whether an account admin ever changes that value. The following SQL can be used to detect changes to the Delta Sharing recipient token lifetime for a metastore within the last 24 hours.", + "query": "SELECT event_time, user_identity.email, account_id, request_params.metastore_id, request_params.delta_sharing_recipient_token_lifetime_in_seconds, response.status_code, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND action_name = 'updateMetastore' AND request_params.delta_sharing_recipient_token_lifetime_in_seconds IS NOT NULL GROUP BY 1, 2, 3, 4, 5, 6 ORDER BY total DESC", + "parent": "system_tables/audit/unity_catalog/queries/", + "alert": { + "name": "delta_sharing_recipient_token_lifetime_change", + "options": { + "column": "total", + "custom_body": "

There have been the following changes to a Delta Sharing recipient token lifetime within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/unity_catalog/alerts/" + } + }, + { + "name": "most_popular_data_products_last_90_days", + "description": "Databricks Unity Catalog is the industry’s first unified governance solution for data and AI on the lakehouse. The main benefit of this unification is that you can define once and secure everywhere, but it also means that appropriately privileged users can report on the most popular data products across an organisation. The following SQL query will show you the most popular data assets by number of requests over the last 90 days.", + "query": "SELECT * FROM (SELECT CASE WHEN isnotnull(request_params.table_full_name) THEN request_params.table_full_name WHEN isnotnull(request_params.volume_full_name) THEN request_params.volume_full_name WHEN isnotnull(request_params.name) THEN request_params.name WHEN isnotnull(request_params.url) THEN request_params.url WHEN isnotnull(request_params.table_url) THEN request_params.table_url WHEN isnotnull(request_params.table_id) THEN request_params.table_id WHEN isnotnull(request_params.volume_id) THEN request_params.volume_id ELSE NULL END AS securable, CASE WHEN isnotnull(request_params.table_full_name) THEN 'TABLE' WHEN isnotnull(request_params.volume_full_name) THEN 'VOLUME' WHEN isnotnull(request_params.share) THEN 'DELTA_SHARE' WHEN isnotnull(request_params.url) THEN 'EXTERNAL_LOCATION' WHEN isnotnull(request_params.table_url) THEN 'TABLE' WHEN isnotnull(request_params.table_id) THEN 'TABLE' WHEN isnotnull(request_params.volume_id) THEN 'VOLUME' ELSE NULL END AS securable_type, count(*) AS total_requests FROM system.access.audit WHERE action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') AND response.status_code = 200 AND event_date >= current_date() - INTERVAL 90 DAYS GROUP BY 1, 2) WHERE NOT startswith(securable, '__databricks_internal.') ORDER BY total_requests DESC LIMIT 500", + "parent": "system_tables/audit/unity_catalog/queries/" + }, + { + "name": "most_privileged_users", + "description": "Identifying our most privileged users can help us to take a risk based approach to security. The following SQL query will provide a relatively simple view of our most privileged users, by showing those with the highest number of different grants to each securable type.", + "query": "WITH catalog_privileges AS (SELECT grantee, 'catalog' AS securable_type, count(*) AS total FROM system.information_schema.catalog_privileges GROUP BY 1, 2), external_location_privileges AS (SELECT grantee, 'external_location' AS securable_type, count(*) AS total FROM system.information_schema.external_location_privileges GROUP BY 1, 2), metastore_privileges AS (SELECT grantee, 'metastore' AS securable_type, count(*) AS total FROM system.information_schema.metastore_privileges GROUP BY 1, 2), routine_privileges AS (SELECT grantee, 'function' AS securable_type, count(*) AS total FROM system.information_schema.routine_privileges GROUP BY 1, 2), schema_privileges AS (SELECT grantee, 'schema' AS securable_type, count(*) AS total FROM system.information_schema.schema_privileges GROUP BY 1, 2), storage_credential_privileges AS (SELECT grantee, 'storage_credential' AS securable_type, count(*) AS total FROM system.information_schema.storage_credential_privileges GROUP BY 1, 2), table_privileges AS (SELECT grantee, 'table' AS securable_type, count(*) AS total FROM system.information_schema.table_privileges GROUP BY 1, 2), volume_privileges AS (SELECT grantee, 'volume' AS securable_type, count(*) AS total FROM system.information_schema.volume_privileges GROUP BY 1, 2) SELECT grantee, securable_type, SUM(totaL) AS number_of_grants FROM (SELECT * FROM catalog_privileges UNION ALL SELECT * FROM external_location_privileges UNION ALL SELECT * FROM metastore_privileges UNION ALL SELECT * FROM routine_privileges UNION ALL SELECT * FROM schema_privileges UNION ALL SELECT * FROM storage_credential_privileges UNION ALL SELECT * FROM table_privileges UNION ALL SELECT * FROM volume_privileges) GROUP BY 1, 2 ORDER BY number_of_grants DESC", + "parent": "system_tables/audit/unity_catalog/queries/" + }, + { + "name": "ip_addresses_used_to_access_uc_data", + "description": "The following SQL query will show you the IP addresses used to access Unity Catalog securables ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') actions over the last 90 days.", + "query": "SELECT regexp_replace(source_ip_address, '(:\\\\d*)', '') AS source_ip_address, CASE WHEN isnotnull(request_params.table_full_name) THEN 'TABLE' WHEN isnotnull(request_params.volume_full_name) THEN 'VOLUME' WHEN isnotnull(request_params.url) THEN 'EXTERNAL_LOCATION' WHEN isnotnull(request_params.table_url) THEN 'TABLE' WHEN isnotnull(request_params.table_id) THEN 'TABLE' WHEN isnotnull(request_params.volume_id) THEN 'VOLUME' WHEN isnotnull(request_params.share) THEN 'DELTA_SHARE' ELSE NULL END AS securable_type, count(*) AS total_requests FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND source_ip_address NOT IN ('', '0.0.0.0', '127.0.0.1') AND action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') GROUP BY 1, 2 ORDER BY total_requests DESC", + "parent": "system_tables/audit/unity_catalog/queries/" + }, + { + "name": "ip_address_ranges_used_to_access_uc_data", + "description": "The following SQL query will show you the IP addresse ranges used to access Unity Catalog securables ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') actions over the last 90 days.", + "query": "SELECT concat(substring_index(source_ip_address, '.', 3), '.0/24') AS source_ip_range, CASE WHEN isnotnull(request_params.table_full_name) THEN 'TABLE' WHEN isnotnull(request_params.volume_full_name) THEN 'VOLUME' WHEN isnotnull(request_params.url) THEN 'EXTERNAL_LOCATION' WHEN isnotnull(request_params.table_url) THEN 'TABLE' WHEN isnotnull(request_params.table_id) THEN 'TABLE' WHEN isnotnull(request_params.volume_id) THEN 'VOLUME' WHEN isnotnull(request_params.share) THEN 'DELTA_SHARE' ELSE NULL END AS securable_type, count(*) AS total_requests FROM system.access.audit WHERE event_date >= current_date() - INTERVAL 90 DAYS AND source_ip_address NOT IN ('', '0.0.0.0', '127.0.0.1') AND action_name IN ('generateTemporaryTableCredential', 'generateTemporaryPathCredential', 'generateTemporaryVolumeCredential', 'deltaSharingQueryTable', 'deltaSharingQueryTableChanges') GROUP BY 1, 2 ORDER BY total_requests DESC", + "parent": "system_tables/audit/unity_catalog/queries/" + }, + { + "name": "clam_av_infected_files_detected", + "description": "Customers using one of our compliance security profile offerings have additional monitoring agents including antivirus installed on their data plane hosts. The following query can be used to detect all antivirus scan events during which infected files have been detected within the last 24 hours. Note that this SQL query/alert will trigger when the ClamAV scan has completed, which may be several hours after the infected file has been found. See clam_av_infected_files_found for a query/alert that will trigger as soon as an infected file has been found.", + "query": "SELECT event_time, workspace_id, request_params.instanceId, regexp_extract(response.result, ('Infected files: (\\\\d+)')) AS infected_files FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'clamAVScanService-dataplane' AND startswith(response.result, 'Infected files:') AND regexp_extract(response.result, ('Infected files: (\\\\d+)')) > 0 ORDER BY event_time DESC", + "parent": "system_tables/audit/compliance_security_profile/queries/", + "alert": { + "name": "clam_av_infected_files_detected", + "options": { + "column": "infected_files", + "custom_body": "

There have been the following infected files detected by ClamAV within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/compliance_security_profile/alerts/" + } + }, + { + "name": "clam_av_infected_files_found", + "description": "Customers using one of our compliance security profile offerings have additional monitoring agents including antivirus installed on their data plane hosts. The following query can be used to detect all antivirus scan events during which infected files have been found within the last 24 hours. Note, that this query/alert will detect infected files as soon as they have been found, rather than when the ClamAV scan finishes.", + "query": "SELECT event_time, workspace_id, request_params.instanceId, regexp_extract(response.result, ': (.*) FOUND') AS signature, regexp_extract(response.result, '(.*):') AS file_path FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'clamAVScanService-dataplane' AND contains(response.result, 'FOUND') ORDER BY event_time DESC", + "parent": "system_tables/audit/compliance_security_profile/queries/", + "alert": { + "name": "clam_av_infected_files_found", + "options": { + "column": "infected_files", + "custom_body": "

There have been the following infected files found by ClamAV within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/compliance_security_profile/alerts/" + } + }, + { + "name": "capsule8_container_breakout_events", + "description": "User code runs in low-privileged containers. A container escape could compromise the security of the cluster especially when running with user isolation for Unity Catalog or Table ACLs. Capsule8 provides a few alerts related to container isolation issues that should be investigated if triggered. The following query can be used to detect all container breakout events within the last 24 hours.", + "query": "SELECT event_time, workspace_id, request_params.instanceId, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'capsule8-alerts-dataplane' AND action_name in ('Container Escape via Kernel Exploitation', 'Userland Container Escape', 'New File Executed in Container', 'Privileged Container Launched') GROUP BY 1, 2, 3 ORDER BY event_time DESC", + "parent": "system_tables/audit/compliance_security_profile/queries/", + "alert": { + "name": "capsule8_container_breakout_events", + "options": { + "column": "total", + "custom_body": "

There have been the following container breakout events detected by Capsule8 within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/compliance_security_profile/alerts/" + } + }, + { + "name": "capsule8_changes_to_host_security_settings", + "description": "No untrusted code or end-user commands should be running on the host OS. There should be no process making changes to security configurations of the host VM. The following SQL query can be used to help us identify suspicious changes within the last 24 hours.", + "query": "SELECT event_time, workspace_id, request_params.instanceId, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'capsule8-alerts-dataplane' AND action_name in ('Processor-Level Protections Disabled', 'AppArmor Disabled In Kernel', 'AppArmor Profile Modified', 'Boot Files Modified', 'Root Certificate Store Modified') GROUP BY 1, 2, 3 ORDER BY event_time DESC", + "parent": "system_tables/audit/compliance_security_profile/queries/", + "alert": { + "name": "capsule8_changes_to_host_security_settings", + "options": { + "column": "total", + "custom_body": "

There have been the following changes to host security settings detected by Capsule8 within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/compliance_security_profile/alerts/" + } + }, + { + "name": "capsule8_kernel_related_events", + "description": "Kernel related events could be another indicator of malicious code running on the host. In particular there should be no kernel modules loaded or internal kernel functions being called by user code. The following SQL query can be used to detect any kernel related events within the last 24 hours.", + "query": "SELECT event_time, workspace_id, request_params.instanceId, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'capsule8-alerts-dataplane' AND action_name in ('BPF Program Executed', 'Kernel Module Loaded', 'Kernel Exploit') GROUP BY 1, 2, 3 ORDER BY event_time DESC", + "parent": "system_tables/audit/compliance_security_profile/queries/", + "alert": { + "name": "capsule8_kernel_related_events", + "options": { + "column": "total", + "custom_body": "

There have been the following kernel related events detected by Capsule8 within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/compliance_security_profile/alerts/" + } + }, + { + "name": "capsule8_suspicious_host_activity", + "description": "Given the architecture of the Databricks containerized runtime and host OS model, only trusted code should be making changes or executing on the host EC2. Changes to containers, evasive actions, or interactive shells could be due to suspicious activity on the host and should be reviewed. The following SQL query can be used to detect suspicious host activity within the last 24 hours.", + "query": "SELECT event_time, workspace_id, request_params.instanceId, count(*) AS total FROM system.access.audit WHERE event_time >= current_timestamp() - INTERVAL 24 HOURS AND service_name = 'capsule8-alerts-dataplane' AND action_name in ('New File Executed in Container', 'Suspicious Interactive Shell', 'User Command Logging Evasion', 'Privileged Container Launched') GROUP BY 1, 2, 3 ORDER BY event_time DESC", + "parent": "system_tables/audit/compliance_security_profile/queries/", + "alert": { + "name": "capsule8_suspicious_host_activity", + "options": { + "column": "total", + "custom_body": "

There have been the following suspicious host activity events detected by Capsule8 within the last 24 hours:


{{QUERY_RESULT_TABLE}}
Link to query
Link to alert", + "custom_subject": "Alert {{ALERT_NAME}} changed status to {{ALERT_STATUS}} because the number of unexpected events is {{ALERT_CONDITION}} than {{ALERT_THRESHOLD}}", + "muted": false, + "op": ">", + "value": "0" + }, + "rearm": "3600", + "parent": "system_tables/audit/compliance_security_profile/alerts/" + } + } + ] + } \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/sql.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/sql.tf new file mode 100644 index 0000000..18eb393 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/sql.tf @@ -0,0 +1,48 @@ +locals { + warehouse_id = var.warehouse_id == "" ? databricks_sql_endpoint.this[0].id : data.databricks_sql_warehouse.this[0].id + data_source_id = var.warehouse_id == "" ? databricks_sql_endpoint.this[0].data_source_id : data.databricks_sql_warehouse.this[0].data_source_id +} + +resource "databricks_sql_endpoint" "this" { + count = var.warehouse_id == "" ? 1 : 0 + warehouse_type = "PRO" + name = "System Tables" + cluster_size = "Small" + max_num_clusters = 1 + auto_stop_mins = 10 +} + +data "databricks_sql_warehouse" "this" { + count = var.warehouse_id == "" ? 0 : 1 + id = var.warehouse_id +} + +resource "databricks_sql_query" "query" { + for_each = local.queries + data_source_id = local.data_source_id + name = local.data_map[each.value].name + query = local.data_map[each.value].query + description = local.data_map[each.value].description + parent = "folders/${databricks_directory.this[local.data_map[each.value].parent].object_id}" + + tags = [ + "system-tables", + ] +} + +resource "databricks_sql_alert" "alert" { + for_each = local.alerts + query_id = databricks_sql_query.query[each.value].id + name = local.data_map[each.value].alert.name + parent = "folders/${databricks_directory.this[local.data_map[each.value].alert.parent].object_id}" + rearm = local.data_map[each.value].alert.rearm + + options { + column = local.data_map[each.value].alert.options.column + op = local.data_map[each.value].alert.options.op + value = local.data_map[each.value].alert.options.value + muted = local.data_map[each.value].alert.options.muted + custom_body = local.data_map[each.value].alert.options.custom_body + custom_subject = local.data_map[each.value].alert.options.custom_subject + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf new file mode 100644 index 0000000..f684f88 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf @@ -0,0 +1,10 @@ +variable "alert_emails" { + type = list(string) + description = "List of emails to notify when alerts are fired" +} + +variable "warehouse_id" { + type = string + default = "" + description = "Optional Warehouse ID to run queries on. If not provided, new SQL Warehouse is created" +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/admin_configuration/admin_configuration.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/admin_configuration/admin_configuration.tf new file mode 100644 index 0000000..72e6bd1 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/admin_configuration/admin_configuration.tf @@ -0,0 +1,14 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/workspace_conf + +resource "databricks_workspace_conf" "just_config_map" { + custom_config = { + "enableResultsDownloading" = "false", // https://docs.databricks.com/en/notebooks/notebook-outputs.html#download-results + "enableNotebookTableClipboard" = "false", // https://docs.databricks.com/en/administration-guide/workspace-settings/notebooks.html#enable-users-to-copy-data-to-the-clipboard-from-notebooks + "enableVerboseAuditLogs" = "true", // https://docs.databricks.com/en/administration-guide/account-settings/verbose-logs.html + "enableDbfsFileBrowser" = "false", // https://docs.databricks.com/en/administration-guide/workspace-settings/dbfs-browser.html + "enableExportNotebook" = "false", // https://docs.databricks.com/en/administration-guide/workspace-settings/notebooks.html#enable-users-to-export-notebooks + "enforceUserIsolation" = "true", // https://docs.databricks.com/en/administration-guide/workspace-settings/enforce-user-isolation.html + "storeInteractiveNotebookResultsInCustomerAccount" = "true", // https://docs.databricks.com/en/administration-guide/workspace-settings/notebooks.html#manage-where-notebook-results-are-stored + "enableUploadDataUis" = "false" // https://docs.databricks.com/en/ingestion/add-data/index.html + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/admin_configuration/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/admin_configuration/provider.tf new file mode 100644 index 0000000..bdd3474 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/admin_configuration/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf new file mode 100644 index 0000000..ed3d319 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf @@ -0,0 +1,79 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster + +// Cluster Version +data "databricks_spark_version" "latest_lts" { + long_term_support = true +} + +// Example Cluster Policy +locals { + default_policy = { + "dbus_per_hour" : { + "type" : "range", + "maxValue" : 10 + }, + "autotermination_minutes" : { + "type" : "fixed", + "value" : 10, + "hidden" : true + }, + "custom_tags.Project" : { + "type" : "fixed", + "value" : var.resource_prefix + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionURL" : null, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionDriverName" : null, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionUserName" : null, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionPassword" : null + } + + isolated_policy = merge( + local.default_policy, + { + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionURL" : { + "type" : "fixed", + "value" : "jdbc:derby:memory:myInMemDB;create=true" + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionDriverName" : { + "type" : "fixed", + "value" : "org.apache.derby.jdbc.EmbeddedDriver" + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionUserName" : { + "type" : "fixed", + "value" : "" + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionPassword" : { + "type" : "fixed", + "value" : "" + } + } + ) + + selected_policy = var.operation_mode == "isolated" ? local.isolated_policy : local.default_policy + + final_policy = { for k, v in local.selected_policy : k => v if v != null } +} + +resource "databricks_cluster_policy" "example" { + name = "Example Cluster Policy" + definition = jsonencode(local.final_policy) +} + +// Cluster Creation +resource "databricks_cluster" "example" { + cluster_name = "Shared Cluster" + data_security_mode = "USER_ISOLATION" + spark_version = data.databricks_spark_version.latest_lts.id + node_type_id = var.compliance_security_profile_egress_ports ? "i3en.xlarge" : "i3.xlarge" + policy_id = databricks_cluster_policy.example.id + autotermination_minutes = 10 + + autoscale { + min_workers = 1 + max_workers = 2 + } + + depends_on = [ + databricks_cluster_policy.example + ] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/provider.tf new file mode 100644 index 0000000..bdd3474 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf new file mode 100644 index 0000000..29344f3 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf @@ -0,0 +1,12 @@ +variable "compliance_security_profile_egress_ports" { + type = bool + nullable = false +} + +variable "operation_mode" { + type = string +} + +variable "resource_prefix" { + type = string +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/ip_access_list.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/ip_access_list.tf new file mode 100644 index 0000000..658fd89 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/ip_access_list.tf @@ -0,0 +1,14 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/ip_access_list + +resource "databricks_workspace_conf" "this" { + custom_config = { + "enableIpAccessLists" = true + } +} + +resource "databricks_ip_access_list" "allowed-list" { + label = "allow_in" + list_type = "ALLOW" + ip_addresses = var.ip_addresses + depends_on = [databricks_workspace_conf.this] +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/provider.tf new file mode 100644 index 0000000..bdd3474 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/variables.tf new file mode 100644 index 0000000..4e1d552 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/ip_access_list/variables.tf @@ -0,0 +1,3 @@ +variable "ip_addresses" { + type = list(string) +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/provider.tf new file mode 100644 index 0000000..1d847d2 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/public_preview/system_schema/system_schema.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/system_schema.tf similarity index 86% rename from aws/tf/modules/sra/databricks_workspace/public_preview/system_schema/system_schema.tf rename to aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/system_schema.tf index a95cda5..180bf9f 100644 --- a/aws/tf/modules/sra/databricks_workspace/public_preview/system_schema/system_schema.tf +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/system_schema.tf @@ -12,10 +12,14 @@ resource "databricks_system_schema" "compute" { schema = "compute" } +resource "databricks_system_schema" "lakeflow" { + schema = "lakeflow" +} + resource "databricks_system_schema" "marketplace" { schema = "marketplace" } resource "databricks_system_schema" "storage" { schema = "storage" -} \ No newline at end of file +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/provider.tf new file mode 100644 index 0000000..1d847d2 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf new file mode 100644 index 0000000..b2491d6 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf @@ -0,0 +1,217 @@ +resource "null_resource" "previous" {} + +resource "time_sleep" "wait_30_seconds" { + depends_on = [null_resource.previous] + + create_duration = "30s" +} + +// Unity Catalog Trust Policy - Data Source +data "aws_iam_policy_document" "passrole_for_unity_catalog_catalog" { + statement { + effect = "Allow" + actions = ["sts:AssumeRole"] + principals { + identifiers = ["arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:role/unity-catalog-prod-UCMasterRole-${var.uc_master_role_id[var.databricks_gov_shard]}"] + type = "AWS" + } + condition { + test = "StringEquals" + variable = "sts:ExternalId" + values = [var.databricks_account_id] + } + } + statement { + sid = "ExplicitSelfRoleAssumption" + effect = "Allow" + actions = ["sts:AssumeRole"] + principals { + type = "AWS" + identifiers = ["arn:aws-us-gov:iam::${var.aws_account_id}:root"] + } + condition { + test = "ArnLike" + variable = "aws:PrincipalArn" + values = ["arn:aws-us-gov:iam::${var.aws_account_id}:role/${var.resource_prefix}-unity-catalog-${var.workspace_id}"] + } + condition { + test = "StringEquals" + variable = "sts:ExternalId" + values = [var.databricks_account_id] + } + } +} + +// Unity Catalog Role +resource "aws_iam_role" "unity_catalog_role" { + name = "${var.resource_prefix}-catalog-${var.workspace_id}" + assume_role_policy = data.aws_iam_policy_document.passrole_for_unity_catalog_catalog.json + tags = { + Name = "${var.resource_prefix}-catalog-${var.workspace_id}" + Project = var.resource_prefix + } +} + +// Unity Catalog IAM Policy +data "aws_iam_policy_document" "unity_catalog_iam_policy" { + statement { + actions = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:PutObject", + "s3:PutObjectAcl", + "s3:DeleteObject", + "s3:ListBucket", + "s3:GetBucketLocation" + ] + + resources = [ + "arn:aws-us-gov:s3:::${var.uc_catalog_name}/*", + "arn:aws-us-gov:s3:::${var.uc_catalog_name}" + ] + + effect = "Allow" + } + + statement { + actions = ["sts:AssumeRole"] + resources = ["arn:aws-us-gov:iam::${var.aws_account_id}:role/${var.resource_prefix}-unity-catalog-${var.workspace_id}"] + effect = "Allow" + } +} + +// Unity Catalog Policy +resource "aws_iam_role_policy" "unity_catalog" { + name = "${var.resource_prefix}-catalog-policy-${var.workspace_id}" + role = aws_iam_role.unity_catalog_role.id + policy = data.aws_iam_policy_document.unity_catalog_iam_policy.json +} + +// Unity Catalog KMS +resource "aws_kms_key" "catalog_storage" { + description = "KMS key for Databricks catalog storage ${var.workspace_id}" + policy = jsonencode({ + Version : "2012-10-17", + "Id" : "key-policy-catalog-storage-${var.workspace_id}", + Statement : [ + { + "Sid" : "Enable IAM User Permissions", + "Effect" : "Allow", + "Principal" : { + "AWS" : [var.cmk_admin_arn] + }, + "Action" : "kms:*", + "Resource" : "*" + }, + { + "Sid" : "Allow IAM Role to use the key", + "Effect" : "Allow", + "Principal" : { + "AWS" : "arn:aws-us-gov:iam::${var.aws_account_id}:role/${var.resource_prefix}-catalog-${var.workspace_id}" + }, + "Action" : [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey*" + ], + "Resource" : "*" + } + ] + }) + tags = { + Name = "${var.resource_prefix}-catalog-storage-${var.workspace_id}-key" + Project = var.resource_prefix + } +} + +resource "aws_kms_alias" "catalog_storage_key_alias" { + name = "alias/${var.resource_prefix}-catalog-storage-${var.workspace_id}-key" + target_key_id = aws_kms_key.catalog_storage.id +} + +// Unity Catalog S3 +resource "aws_s3_bucket" "unity_catalog_bucket" { + bucket = var.uc_catalog_name + force_destroy = true + tags = { + Name = var.uc_catalog_name + Project = var.resource_prefix + } +} + +resource "aws_s3_bucket_versioning" "unity_catalog_versioning" { + bucket = aws_s3_bucket.unity_catalog_bucket.id + versioning_configuration { + status = "Disabled" + } +} + +resource "aws_s3_bucket_server_side_encryption_configuration" "unity_catalog" { + bucket = aws_s3_bucket.unity_catalog_bucket.bucket + + rule { + bucket_key_enabled = true + apply_server_side_encryption_by_default { + sse_algorithm = "aws:kms" + kms_master_key_id = aws_kms_key.catalog_storage.arn + } + } + depends_on = [aws_kms_alias.catalog_storage_key_alias] +} + +resource "aws_s3_bucket_public_access_block" "unity_catalog" { + bucket = aws_s3_bucket.unity_catalog_bucket.id + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true + depends_on = [aws_s3_bucket.unity_catalog_bucket] +} + +// Storage Credential +resource "databricks_storage_credential" "workspace_catalog_storage_credential" { + name = aws_iam_role.unity_catalog_role.name + aws_iam_role { + role_arn = aws_iam_role.unity_catalog_role.arn + } + depends_on = [aws_iam_role.unity_catalog_role, time_sleep.wait_30_seconds] + isolation_mode = "ISOLATION_MODE_ISOLATED" +} + +// External Location +resource "databricks_external_location" "workspace_catalog_external_location" { + name = var.uc_catalog_name + url = "s3://${var.uc_catalog_name}/catalog/" + credential_name = databricks_storage_credential.workspace_catalog_storage_credential.id + skip_validation = true + read_only = false + comment = "External location for catalog ${var.uc_catalog_name}" + isolation_mode = "ISOLATION_MODE_ISOLATED" +} + +// Workspace Catalog +resource "databricks_catalog" "workspace_catalog" { + name = replace(var.uc_catalog_name, "-", "_") + comment = "This catalog is for workspace - ${var.workspace_id}" + isolation_mode = "ISOLATED" + storage_root = "s3://${var.uc_catalog_name}/catalog/" + properties = { + purpose = "Catalog for workspace - ${var.workspace_id}" + } + depends_on = [databricks_external_location.workspace_catalog_external_location] +} + +// Set Workspace Catalog as Default +resource "databricks_default_namespace_setting" "this" { + namespace { + value = replace(var.uc_catalog_name, "-", "_") + } +} + +// Grant Admin Catalog Perms +resource "databricks_grant" "workspace_catalog" { + catalog = databricks_catalog.workspace_catalog.name + + principal = var.user_workspace_catalog_admin + privileges = ["ALL_PRIVILEGES"] +} diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf new file mode 100644 index 0000000..411a886 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf @@ -0,0 +1,39 @@ +variable "aws_account_id" { + type = string +} + +variable "cmk_admin_arn" { + type = string +} + +variable "databricks_account_id" { + type = string +} + +variable "databricks_gov_shard" { + type = string +} + +variable "databricks_prod_aws_account_id" { + type = map(string) +} + +variable "resource_prefix" { + type = string +} + +variable "uc_catalog_name" { + type = string +} + +variable "uc_master_role_id" { + type = map(string) +} + +variable "user_workspace_catalog_admin" { + type = string +} + +variable "workspace_id" { + type = string +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/provider.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/provider.tf new file mode 100644 index 0000000..1d847d2 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf new file mode 100644 index 0000000..3404dab --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf @@ -0,0 +1,116 @@ +resource "null_resource" "previous" {} + +resource "time_sleep" "wait_30_seconds" { + depends_on = [null_resource.previous] + + create_duration = "30s" +} + +// Storage Credential Trust Policy +data "aws_iam_policy_document" "passrole_for_storage_credential" { + statement { + effect = "Allow" + actions = ["sts:AssumeRole"] + principals { + identifiers = ["arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:role/unity-catalog-prod-UCMasterRole-${var.uc_master_role_id[var.databricks_gov_shard]}"] + type = "AWS" + } + condition { + test = "StringEquals" + variable = "sts:ExternalId" + values = [var.databricks_account_id] + } + } + statement { + sid = "ExplicitSelfRoleAssumption" + effect = "Allow" + actions = ["sts:AssumeRole"] + principals { + type = "AWS" + identifiers = ["arn:aws-us-gov:iam::${var.aws_account_id}:root"] + } + condition { + test = "ArnLike" + variable = "aws:PrincipalArn" + values = ["arn:aws-us-gov:iam::${var.aws_account_id}:role/${var.resource_prefix}-storage-credential"] + } + condition { + test = "StringEquals" + variable = "sts:ExternalId" + values = [var.databricks_account_id] + } + } +} + +// Storage Credential Role +resource "aws_iam_role" "storage_credential_role" { + name = "${var.resource_prefix}-storage-credential-example" + assume_role_policy = data.aws_iam_policy_document.passrole_for_storage_credential.json + tags = { + Name = "${var.resource_prefix}-storage-credential-example" + Project = var.resource_prefix + } +} + +// Storage Credential Policy +resource "aws_iam_role_policy" "storage_credential_policy" { + name = "${var.resource_prefix}-storage-credential-policy-example" + role = aws_iam_role.storage_credential_role.id + policy = jsonencode({ + Version : "2012-10-17", + Statement : [ + { + "Action" : [ + "s3:GetObject", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:GetLifecycleConfiguration", + ], + "Resource" : [ + "arn:aws-us-gov:s3:::${var.read_only_data_bucket}/*", + "arn:aws-us-gov:s3:::${var.read_only_data_bucket}" + ], + "Effect" : "Allow" + }, + { + "Action" : [ + "sts:AssumeRole" + ], + "Resource" : [ + "arn:aws-us-gov:iam::${var.aws_account_id}:role/${var.resource_prefix}-storage-credential-example" + ], + "Effect" : "Allow" + } + ] + } + ) +} + +// Storage Credential +resource "databricks_storage_credential" "external" { + name = aws_iam_role.storage_credential_role.name + aws_iam_role { + role_arn = aws_iam_role.storage_credential_role.arn + } + isolation_mode = "ISOLATION_MODE_ISOLATED" + depends_on = [aws_iam_role.storage_credential_role, time_sleep.wait_30_seconds] +} + +// External Location +resource "databricks_external_location" "data_example" { + name = "external-location-example" + url = "s3://${var.read_only_data_bucket}/" + credential_name = databricks_storage_credential.external.id + read_only = true + comment = "Read only external location for ${var.read_only_data_bucket}" + isolation_mode = "ISOLATION_MODE_ISOLATED" +} + +// External Location Grant +resource "databricks_grants" "data_example" { + external_location = databricks_external_location.data_example.id + grant { + principal = var.read_only_external_location_admin + privileges = ["ALL_PRIVILEGES"] + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf new file mode 100644 index 0000000..fa0b336 --- /dev/null +++ b/aws-gov/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf @@ -0,0 +1,31 @@ +variable "aws_account_id" { + type = string +} + +variable "databricks_account_id" { + type = string +} + +variable "databricks_gov_shard" { + type = string +} + +variable "databricks_prod_aws_account_id" { + type = map(string) +} + +variable "read_only_data_bucket" { + type = string +} + +variable "read_only_external_location_admin" { + type = string +} + +variable "resource_prefix" { + type = string +} + +variable "uc_master_role_id" { + type = map(string) +} diff --git a/aws-gov/tf/modules/sra/network.tf b/aws-gov/tf/modules/sra/network.tf new file mode 100644 index 0000000..b4333ca --- /dev/null +++ b/aws-gov/tf/modules/sra/network.tf @@ -0,0 +1,89 @@ +// EXPLANATION: Create the customer managed-vpc and security group rules + +// VPC and other assets - skipped entirely in custom mode, some assets skipped for firewall and isolated +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + version = "5.1.1" + + count = var.operation_mode != "custom" ? 1 : 0 + + name = "${var.resource_prefix}-classic-compute-plane-vpc" + cidr = var.vpc_cidr_range + azs = var.availability_zones + + enable_dns_hostnames = true + enable_nat_gateway = var.operation_mode == "firewall" || var.operation_mode == "isolated" ? false : true + single_nat_gateway = false + one_nat_gateway_per_az = var.operation_mode == "firewall" || var.operation_mode == "isolated" ? false : true + create_igw = var.operation_mode == "firewall" || var.operation_mode == "isolated" ? false : true + + public_subnet_names = var.operation_mode == "firewall" || var.operation_mode == "isolated" ? [] : [for az in var.availability_zones : format("%s-public-%s", var.resource_prefix, az)] + public_subnets = var.operation_mode == "firewall" || var.operation_mode == "isolated" ? [] : var.public_subnets_cidr + + private_subnet_names = [for az in var.availability_zones : format("%s-private-%s", var.resource_prefix, az)] + private_subnets = var.private_subnets_cidr + + intra_subnet_names = [for az in var.availability_zones : format("%s-privatelink-%s", var.resource_prefix, az)] + intra_subnets = var.privatelink_subnets_cidr + + tags = { + Project = var.resource_prefix + } +} + +// Security group - skipped in custom mode +resource "aws_security_group" "sg" { + count = var.operation_mode != "custom" ? 1 : 0 + + vpc_id = module.vpc[0].vpc_id + depends_on = [module.vpc] + + dynamic "ingress" { + for_each = ["tcp", "udp"] + content { + description = "Databricks - Workspace SG - Internode Communication" + from_port = 0 + to_port = 65535 + protocol = ingress.value + self = true + } + } + + dynamic "egress" { + for_each = ["tcp", "udp"] + content { + description = "Databricks - Workspace SG - Internode Communication" + from_port = 0 + to_port = 65535 + protocol = egress.value + self = true + } + } + + dynamic "egress" { + for_each = var.sg_egress_ports + content { + description = "Databricks - Workspace SG - REST (443), Secure Cluster Connectivity (6666), Future Extendability (8443-8451)" + from_port = egress.value + to_port = egress.value + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + } + + dynamic "egress" { + for_each = var.compliance_security_profile_egress_ports ? [2443] : [] + + content { + description = "Databricks - Workspace Security Group - FIPS encryption" + from_port = 2443 + to_port = 2443 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + } + tags = { + Name = "${var.resource_prefix}-workspace-sg" + Project = var.resource_prefix + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/outputs.tf b/aws-gov/tf/modules/sra/outputs.tf new file mode 100644 index 0000000..0ac5ce9 --- /dev/null +++ b/aws-gov/tf/modules/sra/outputs.tf @@ -0,0 +1,3 @@ +output "databricks_host" { + value = module.databricks_mws_workspace.workspace_url +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/privatelink.tf b/aws-gov/tf/modules/sra/privatelink.tf new file mode 100644 index 0000000..d85dff3 --- /dev/null +++ b/aws-gov/tf/modules/sra/privatelink.tf @@ -0,0 +1,324 @@ +// Security group for privatelink - skipped in custom operation mode +resource "aws_security_group" "privatelink" { + count = var.operation_mode != "custom" ? 1 : 0 + + vpc_id = module.vpc[0].vpc_id + + ingress { + description = "Databricks - PrivateLink Endpoint SG - REST API" + from_port = 443 + to_port = 443 + protocol = "tcp" + security_groups = [aws_security_group.sg[0].id] + } + + ingress { + description = "Databricks - PrivateLink Endpoint SG - Secure Cluster Connectivity" + from_port = 6666 + to_port = 6666 + protocol = "tcp" + security_groups = [aws_security_group.sg[0].id] + } + + ingress { + description = "Databricks - PrivateLink Endpoint SG - Future Extendability" + from_port = 8443 + to_port = 8451 + protocol = "tcp" + security_groups = [aws_security_group.sg[0].id] + } + + dynamic "ingress" { + for_each = var.compliance_security_profile_egress_ports ? [2443] : [] + + content { + description = "Databricks - PrivateLink Endpoint SG - FIPS encryption" + from_port = 2443 + to_port = 2443 + protocol = "tcp" + security_groups = [aws_security_group.sg[0].id] + } + } + + tags = { + Name = "${var.resource_prefix}-private-link-sg", + Project = var.resource_prefix + } +} + +// EXPLANATION: VPC Gateway Endpoint for S3, Interface Endpoint for Kinesis, and Interface Endpoint for STS + + +// Restrictive S3 endpoint policy - only used if restrictive S3 endpoint policy is enabled +data "aws_iam_policy_document" "s3_vpc_endpoint_policy" { + count = var.enable_restrictive_s3_endpoint_boolean ? 1 : 0 + + statement { + sid = "Grant access to Databricks Root Bucket" + effect = "Allow" + actions = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:PutObject", + "s3:DeleteObject", + "s3:ListBucket", + "s3:GetBucketLocation" + ] + + principals { + type = "AWS" + identifiers = ["*"] + } + + resources = [ + "arn:aws-us-gov:s3:::${var.resource_prefix}-workspace-root-storage/*", + "arn:aws-us-gov:s3:::${var.resource_prefix}-workspace-root-storage" + ] + + condition { + test = "StringEquals" + variable = "aws:PrincipalAccount" + values = ["${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}"] + } + + condition { + test = "StringEqualsIfExists" + variable = "aws:SourceVpc" + values = [ + module.vpc[0].vpc_id + ] + } + } + + statement { + sid = "Grant access to Databricks Unity Catalog Metastore Bucket" + effect = "Allow" + actions = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:PutObject", + "s3:DeleteObject", + "s3:ListBucket", + "s3:GetBucketLocation" + ] + + principals { + type = "AWS" + identifiers = ["*"] + } + + resources = [ + "arn:aws-us-gov:s3:::${var.resource_prefix}-catalog-${module.databricks_mws_workspace.workspace_id}/*", + "arn:aws-us-gov:s3:::${var.resource_prefix}-catalog-${module.databricks_mws_workspace.workspace_id}" + ] + } + + statement { + sid = "Grant read-only access to Data Bucket" + effect = "Allow" + actions = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:ListBucket", + "s3:GetBucketLocation" + ] + + principals { + type = "AWS" + identifiers = ["*"] + } + + resources = [ + "arn:aws-us-gov:s3:::${var.read_only_data_bucket}/*", + "arn:aws-us-gov:s3:::${var.read_only_data_bucket}" + ] + } + + statement { + sid = "Grant Databricks Read Access to Artifact and Data Buckets" + effect = "Allow" + actions = [ + "s3:ListBucket", + "s3:GetObjectVersion", + "s3:GetObject", + "s3:GetBucketLocation" + ] + + principals { + type = "AWS" + identifiers = ["*"] + } + + resources = [ + "arn:aws-us-gov:s3:::databricks-prod-artifacts-${var.region}/*", + "arn:aws-us-gov:s3:::databricks-prod-artifacts-${var.region}", + "arn:aws-us-gov:s3:::databricks-datasets-${var.region_name}/*", + "arn:aws-us-gov:s3:::databricks-datasets-${var.region_name}" + ] + } + + statement { + sid = "Grant access to Databricks Log Bucket" + effect = "Allow" + actions = [ + "s3:PutObject", + "s3:ListBucket", + "s3:GetBucketLocation" + ] + + principals { + type = "AWS" + identifiers = ["*"] + } + + resources = [ + "arn:aws-us-gov:s3:::databricks-prod-storage-${var.region_name}/*", + "arn:aws-us-gov:s3:::databricks-prod-storage-${var.region_name}" + ] + + condition { + test = "StringEquals" + variable = "aws:PrincipalAccount" + values = ["${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}"] + } + } + depends_on = [module.databricks_mws_workspace] +} + +// Restrictive STS endpoint policy - only used if restrictive STS endpoint policy is enabled +data "aws_iam_policy_document" "sts_vpc_endpoint_policy" { + count = var.enable_restrictive_sts_endpoint_boolean ? 1 : 0 + + statement { + actions = [ + "sts:AssumeRole", + "sts:GetAccessKeyInfo", + "sts:GetSessionToken", + "sts:DecodeAuthorizationMessage", + "sts:TagSession" + ] + effect = "Allow" + resources = ["*"] + + principals { + type = "AWS" + identifiers = ["${var.aws_account_id}"] + } + } + + statement { + actions = [ + "sts:AssumeRole", + "sts:GetSessionToken", + "sts:TagSession" + ] + effect = "Allow" + resources = ["*"] + + principals { + type = "AWS" + identifiers = [ + "arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:user/databricks-datasets-readonly-user-prod", + "${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}" + ] + } + } +} + +// Restrictive Kinesis endpoint policy - only used if restrictive Kinesis endpoint policy is enabled +data "aws_iam_policy_document" "kinesis_vpc_endpoint_policy" { + count = var.enable_restrictive_kinesis_endpoint_boolean ? 1 : 0 + + statement { + actions = [ + "kinesis:PutRecord", + "kinesis:PutRecords", + "kinesis:DescribeStream" + ] + effect = "Allow" + resources = ["arn:aws-us-gov:kinesis:${var.region}:${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:stream/*"] + + principals { + type = "AWS" + identifiers = ["${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}"] + } + } +} + +// VPC endpoint creation - Skipped in custom operation mode +module "vpc_endpoints" { + count = var.operation_mode != "custom" ? 1 : 0 + + source = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints" + version = "3.11.0" + + vpc_id = module.vpc[0].vpc_id + security_group_ids = [aws_security_group.privatelink[0].id] + + endpoints = { + s3 = { + service = "s3" + service_type = "Gateway" + route_table_ids = module.vpc[0].private_route_table_ids + policy = var.enable_restrictive_s3_endpoint_boolean ? data.aws_iam_policy_document.s3_vpc_endpoint_policy[0].json : null + tags = { + Name = "${var.resource_prefix}-s3-vpc-endpoint" + Project = var.resource_prefix + } + }, + sts = { + service = "sts" + private_dns_enabled = true + subnet_ids = module.vpc[0].intra_subnets + policy = var.enable_restrictive_sts_endpoint_boolean ? data.aws_iam_policy_document.sts_vpc_endpoint_policy[0].json : null + tags = { + Name = "${var.resource_prefix}-sts-vpc-endpoint" + Project = var.resource_prefix + } + }, + kinesis-streams = { + service = "kinesis-streams" + private_dns_enabled = true + subnet_ids = module.vpc[0].intra_subnets + policy = var.enable_restrictive_kinesis_endpoint_boolean ? data.aws_iam_policy_document.kinesis_vpc_endpoint_policy[0].json : null + tags = { + Name = "${var.resource_prefix}-kinesis-vpc-endpoint" + Project = var.resource_prefix + } + } + } +} + +// Databricks REST endpoint - skipped in custom operation mode +resource "aws_vpc_endpoint" "backend_rest" { + count = var.operation_mode != "custom" ? 1 : 0 + + vpc_id = module.vpc[0].vpc_id + service_name = var.workspace[var.databricks_gov_shard] + vpc_endpoint_type = "Interface" + security_group_ids = [aws_security_group.privatelink[0].id] + subnet_ids = module.vpc[0].intra_subnets + private_dns_enabled = true + depends_on = [module.vpc.vpc_id] + tags = { + Name = "${var.resource_prefix}-databricks-backend-rest" + Project = var.resource_prefix + } +} + +// Databricks SCC endpoint - skipped in custom operation mode +resource "aws_vpc_endpoint" "backend_relay" { + count = var.operation_mode != "custom" ? 1 : 0 + + vpc_id = module.vpc[0].vpc_id + service_name = var.scc_relay[var.databricks_gov_shard] + vpc_endpoint_type = "Interface" + security_group_ids = [aws_security_group.privatelink[0].id] + subnet_ids = module.vpc[0].intra_subnets + private_dns_enabled = true + depends_on = [module.vpc.vpc_id] + tags = { + Name = "${var.resource_prefix}-databricks-backend-relay" + Project = var.resource_prefix + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/provider.tf b/aws-gov/tf/modules/sra/provider.tf new file mode 100644 index 0000000..ebaeb4f --- /dev/null +++ b/aws-gov/tf/modules/sra/provider.tf @@ -0,0 +1,21 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + configuration_aliases = [ + databricks.mws + ] + } + aws = { + source = "hashicorp/aws" + } + } +} + +provider "databricks" { + alias = "created_workspace" + host = module.databricks_mws_workspace.workspace_url + account_id = var.databricks_account_id + client_id = var.client_id + client_secret = var.client_secret +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/root_s3_bucket.tf b/aws-gov/tf/modules/sra/root_s3_bucket.tf new file mode 100644 index 0000000..6c1d617 --- /dev/null +++ b/aws-gov/tf/modules/sra/root_s3_bucket.tf @@ -0,0 +1,89 @@ +// EXPLANATION: Create the workspace root bucket + +resource "aws_s3_bucket" "root_storage_bucket" { + bucket = "${var.resource_prefix}-workspace-root-storage" + force_destroy = true + tags = { + Name = "${var.resource_prefix}-workspace-root-storage" + Project = var.resource_prefix + } +} + +resource "aws_s3_bucket_versioning" "root_bucket_versioning" { + bucket = aws_s3_bucket.root_storage_bucket.id + versioning_configuration { + status = "Disabled" + } +} + +resource "aws_s3_bucket_server_side_encryption_configuration" "root_storage_bucket" { + bucket = aws_s3_bucket.root_storage_bucket.bucket + + rule { + bucket_key_enabled = true + apply_server_side_encryption_by_default { + sse_algorithm = "aws:kms" + kms_master_key_id = aws_kms_key.workspace_storage.arn + } + } + depends_on = [aws_kms_alias.workspace_storage_key_alias] +} + +resource "aws_s3_bucket_public_access_block" "root_storage_bucket" { + bucket = aws_s3_bucket.root_storage_bucket.id + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true + depends_on = [aws_s3_bucket.root_storage_bucket] +} + +data "aws_iam_policy_document" "this" { + statement { + effect = "Allow" + actions = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:PutObject", + "s3:DeleteObject"] + resources = [ + "${aws_s3_bucket.root_storage_bucket.arn}/*", + aws_s3_bucket.root_storage_bucket.arn] + principals { + identifiers = ["arn:aws-us-gov:iam::${var.databricks_prod_aws_account_id[var.databricks_gov_shard]}:root"] + type = "AWS" + } + condition { + test = "StringEquals" + variable = "aws:PrincipalTag/DatabricksAccountId" + + values = [ + var.databricks_account_id + ] + } + } +} + +# Bucket policy to use if the restrictive root bucket is set to false +resource "aws_s3_bucket_policy" "root_bucket_policy" { + count = var.enable_restrictive_root_bucket_boolean ? 0 : 1 + + bucket = aws_s3_bucket.root_storage_bucket.id + policy = data.aws_iam_policy_document.this.json + depends_on = [aws_s3_bucket_public_access_block.root_storage_bucket] +} + +# Bucket policy to use if the restrictive root bucket is set to true +resource "aws_s3_bucket_policy" "root_bucket_policy_ignore" { + count = var.enable_restrictive_root_bucket_boolean ? 1 : 0 + + bucket = aws_s3_bucket.root_storage_bucket.id + policy = data.aws_iam_policy_document.this.json + depends_on = [aws_s3_bucket_public_access_block.root_storage_bucket] + + lifecycle { + ignore_changes = [policy] + } +} \ No newline at end of file diff --git a/aws-gov/tf/modules/sra/variables.tf b/aws-gov/tf/modules/sra/variables.tf new file mode 100644 index 0000000..c235c3b --- /dev/null +++ b/aws-gov/tf/modules/sra/variables.tf @@ -0,0 +1,294 @@ +variable "availability_zones" { + description = "List of AWS availability zones." + type = list(string) +} + +variable "aws_account_id" { + description = "ID of the AWS account." + type = string + sensitive = true +} + +variable "client_id" { + description = "Client ID for Databricks authentication." + type = string + sensitive = true +} + +variable "client_secret" { + description = "Secret key for the Databricks client ID." + type = string + sensitive = true +} + +variable "cmk_admin_arn" { + description = "Amazon Resource Name (ARN) of the CMK admin." + type = string +} + +variable "compliance_security_profile_egress_ports" { + type = bool + description = "Add 2443 to security group configuration or nitro instance" + nullable = false +} + +variable "custom_private_subnet_ids" { + type = list(string) + description = "List of custom private subnet IDs" +} + +variable "custom_relay_vpce_id" { + type = string + description = "Custom Relay VPC Endpoint ID" +} + +variable "custom_sg_id" { + type = string + description = "Custom security group ID" +} + +variable "custom_vpc_id" { + type = string + description = "Custom VPC ID" +} + +variable "custom_workspace_vpce_id" { + type = string + description = "Custom Workspace VPC Endpoint ID" +} + +variable "databricks_account_id" { + description = "ID of the Databricks account." + type = string + sensitive = true +} + +variable "enable_admin_configs_boolean" { + type = bool + description = "Enable workspace configs" + nullable = false +} + +variable "enable_audit_log_alerting" { + description = "Flag to audit log alerting." + type = bool + sensitive = true + default = false +} + +variable "enable_cluster_boolean" { + description = "Flag to enable cluster." + type = bool + sensitive = true + default = false +} + +variable "enable_ip_boolean" { + description = "Flag to enable IP-related configurations." + type = bool + sensitive = true + default = false +} + +variable "enable_logging_boolean" { + description = "Flag to enable logging." + type = bool + sensitive = true + default = false +} + +variable "enable_read_only_external_location_boolean" { + description = "Flag to enable read only external location" + type = bool + sensitive = true + default = false +} + +variable "enable_restrictive_kinesis_endpoint_boolean" { + type = bool + description = "Enable restrictive Kinesis endpoint boolean flag" + default = false +} + +variable "enable_restrictive_root_bucket_boolean" { + description = "Flag to enable restrictive root bucket settings." + type = bool + sensitive = true + default = false +} + +variable "enable_restrictive_s3_endpoint_boolean" { + type = bool + description = "Enable restrictive S3 endpoint boolean flag" + default = false +} + +variable "enable_restrictive_sts_endpoint_boolean" { + type = bool + description = "Enable restrictive STS endpoint boolean flag" + default = false +} + +variable "enable_sat_boolean" { + description = "Flag for a specific SAT (Service Access Token) configuration." + type = bool + sensitive = true + default = false +} + +variable "enable_system_tables_schema_boolean" { + description = "Flag for enabling public preview system schema access" + type = bool + sensitive = true + default = false +} + +variable "firewall_allow_list" { + description = "List of allowed firewall rules." + type = list(string) +} + +variable "firewall_subnets_cidr" { + description = "CIDR blocks for firewall subnets." + type = list(string) +} + +variable "hms_fqdn" { + type = map(string) + default = { + "civilian" = "discovery-search-rds-prod-dbdiscoverysearch-uus7j2cyyu1m.c40ji7ukhesx.us-gov-west-1.rds.amazonaws.com" + "dod" = "lineage-usgovwest1dod-prod.cpnejponioft.us-gov-west-1.rds.amazonaws.com" + } +} + +variable "ip_addresses" { + description = "List of IP addresses to allow list." + type = list(string) +} + +variable "metastore_exists" { + description = "If a metastore exists" + type = bool +} + +variable "operation_mode" { + type = string + description = "The type of Operation Mode for the workspace network configuration." + nullable = false + + validation { + condition = contains(["sandbox", "firewall", "custom", "isolated"], var.operation_mode) + error_message = "Invalid operation mode. Allowed values are: sandbox, firewall, custom, isolated." + } +} + +variable "private_subnets_cidr" { + description = "CIDR blocks for private subnets." + type = list(string) +} + +variable "privatelink_subnets_cidr" { + description = "CIDR blocks for private link subnets." + type = list(string) +} + +variable "public_subnets_cidr" { + description = "CIDR blocks for public subnets." + type = list(string) +} + +variable "read_only_data_bucket" { + description = "S3 bucket for data storage." + type = string +} + +variable "read_only_external_location_admin" { + description = "User to grant external location admin." + type = string +} + +variable "region" { + description = "AWS region code." + type = string +} + +variable "region_name" { + description = "Name of the AWS region." + type = string +} + +variable "resource_prefix" { + description = "Prefix for the resource names." + type = string +} + +variable "scc_relay" { + type = map(string) + default = { + "civilian" = "com.amazonaws.vpce.us-gov-west-1.vpce-svc-05f27abef1a1a3faa" + "dod" = "com.amazonaws.vpce.us-gov-west-1.vpce-svc-08fddf710780b2a54" + } +} + +variable "sg_egress_ports" { + description = "List of egress ports for security groups." + type = list(string) +} + +variable "user_workspace_admin" { + description = "User to grant admin workspace access." + type = string + nullable = false +} + +variable "user_workspace_catalog_admin" { + description = "Admin for the workspace catalog" + type = string +} + +variable "vpc_cidr_range" { + description = "CIDR range for the VPC." + type = string +} + +variable "workspace" { + type = map(string) + default = { + "civilian" = "com.amazonaws.vpce.us-gov-west-1.vpce-svc-0f25e28401cbc9418" + "dod" = "com.amazonaws.vpce.us-gov-west-1.vpce-svc-05c210a2feea23ad7" + } +} + +// AWS Gov Only Variables +variable "databricks_gov_shard" { + description = "Gov Shard civilian or dod" + type = string +} + + +variable "databricks_prod_aws_account_id" { + description = "Databricks Prod AWS Account Id" + type = map(string) + default = { + "civilian" = "044793339203" + "dod" = "170661010020" + } +} + +variable "log_delivery_role_name" { + description = "Log Delivery Role Name" + type = map(string) + default = { + "civilian" = "SaasUsageDeliveryRole-prod-aws-gov-IAMRole-L4QM0RCHYQ1G" + "dod" = "SaasUsageDeliveryRole-prod-aws-gov-dod-IAMRole-1DMEHBYR8VC5P" + } +} + +variable "uc_master_role_id" { + description = "UC Master Role ID" + type = map(string) + default = { + "civilian" = "1QRFA8SGY15OJ" + "dod" = "1DI6DL6ZP26AS" + } +} diff --git a/aws-gov/tf/provider.tf b/aws-gov/tf/provider.tf new file mode 100644 index 0000000..66469b3 --- /dev/null +++ b/aws-gov/tf/provider.tf @@ -0,0 +1,28 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + version = " 1.50.0" + } + aws = { + source = "hashicorp/aws" + } + } +} + +provider "aws" { + region = var.region + default_tags { + tags = { + Resource = var.resource_prefix + } + } +} + +provider "databricks" { + alias = "mws" + host = var.account_console[var.databricks_gov_shard] + account_id = var.databricks_account_id + client_id = var.client_id + client_secret = var.client_secret +} \ No newline at end of file diff --git a/aws-gov/tf/sra.tf b/aws-gov/tf/sra.tf new file mode 100644 index 0000000..90af84e --- /dev/null +++ b/aws-gov/tf/sra.tf @@ -0,0 +1,71 @@ +module "SRA" { + source = "./modules/sra" + providers = { + databricks.mws = databricks.mws + aws = aws + } + + // REQUIRED - Authentication: + databricks_account_id = var.databricks_account_id + client_id = var.client_id + client_secret = var.client_secret + aws_account_id = var.aws_account_id + region = var.region + databricks_gov_shard = var.databricks_gov_shard + region_name = var.region_name[var.databricks_gov_shard] + + // REQUIRED - Naming and Tagging: + resource_prefix = var.resource_prefix + + // REQUIRED - Workspace and Unity Catalog: + user_workspace_admin = null // Workspace admin user email. + user_workspace_catalog_admin = null // Workspace catalog admin email. + operation_mode = "isolated" // Operation mode (sandbox, custom, firewall, isolated), see README.md for more information. + metastore_exists = false // If a regional metastore exists set to true. If there are multiple regional metastores, you can comment out "uc_init" and add the metastore ID directly in to the module call for "uc_assignment". + + // REQUIRED - AWS Infrastructure: + cmk_admin_arn = null // CMK admin ARN, defaults to the AWS account root user. + vpc_cidr_range = "10.0.0.0/18" // Please re-define the subsequent subnet ranges if the VPC CIDR range is updated. + private_subnets_cidr = ["10.0.0.0/22", "10.0.4.0/22", "10.0.8.0/22"] + privatelink_subnets_cidr = ["10.0.28.0/26", "10.0.28.64/26", "10.0.28.128/26"] + availability_zones = [data.aws_availability_zones.available.names[0], data.aws_availability_zones.available.names[1], data.aws_availability_zones.available.names[2]] + sg_egress_ports = [443, 3306, 8443, 8444, 8445, 8446, 8447, 8448, 8449, 8450, 8451] + compliance_security_profile_egress_ports = true // Set to true to enable compliance security profile related egress ports (2443) + + // Operation Mode Specific: + // Sandbox and Firewall Operation Mode: + public_subnets_cidr = ["10.0.29.0/26", "10.0.29.64/26", "10.0.29.128/26"] + + // Firewall Operation Mode: + firewall_subnets_cidr = ["10.0.33.0/26", "10.0.33.64/26", "10.0.33.128/26"] + firewall_allow_list = [".pypi.org", ".cran.r-project.org", ".pythonhosted.org", ".spark-packages.org", ".maven.org", "maven.apache.org", ".storage-download.googleapis.com"] + + // Custom Operation Mode: + custom_vpc_id = null + custom_private_subnet_ids = null // List of custom private subnet IDs required. + custom_sg_id = null + custom_relay_vpce_id = null + custom_workspace_vpce_id = null + + // OPTIONAL - Examples, Workspace Hardening, and Solution Accelerators: + enable_read_only_external_location_boolean = false // Set to true to enable a read-only external location. + read_only_data_bucket = null // S3 bucket name for read-only data. + read_only_external_location_admin = null // Admin for the external location. + + enable_cluster_boolean = false // Set to true to create a default Databricks clusters. + enable_admin_configs_boolean = false // Set to true to enable optional admin configurations. + enable_logging_boolean = false // Set to true to enable log delivery and creation of related assets (e.g. S3 bucket and IAM role) + + enable_restrictive_root_bucket_boolean = false // Set to true to enable a restrictive root bucket policy, this is subject to change and may cause unexpected issues in the event of a change. + enable_restrictive_s3_endpoint_boolean = false // Set to true to enable a restrictive S3 endpoint policy, this is subject to change and may cause unexpected issues in the event of a change. + enable_restrictive_sts_endpoint_boolean = false // Set to true to enable a restrictive STS endpoint policy, this is subject to change and may cause unexpected issues in the event of a change. + enable_restrictive_kinesis_endpoint_boolean = false // Set to true to enable a restrictive Kinesis endpoint policy, this is subject to change and may cause unexpected issues in the event of a change. + + enable_ip_boolean = false // Set to true to enable IP access list. + ip_addresses = ["X.X.X.X", "X.X.X.X/XX", "X.X.X.X/XX"] // Specify IP addresses for access. + + enable_system_tables_schema_boolean = false // Set to true to enable system table schemas + + enable_sat_boolean = false // Set to true to enable Security Analysis Tool. https://github.com/databricks-industry-solutions/security-analysis-tool + enable_audit_log_alerting = false // Set to true to create 40+ queries for audit log alerting based on user activity. https://github.com/andyweaves/system-tables-audit-logs +} \ No newline at end of file diff --git a/aws-gov/tf/template.tfvars.example b/aws-gov/tf/template.tfvars.example new file mode 100644 index 0000000..88e026d --- /dev/null +++ b/aws-gov/tf/template.tfvars.example @@ -0,0 +1,8 @@ +# Configuration Variables for AWS and Databricks + +aws_account_id = "" // AWS account ID where resources will be deployed. +client_id = "" // Service principal ID for Databricks with admin permissions. +client_secret = "" // Secret for the corresponding service principal. +databricks_account_id = "" // Databricks account ID. +databricks_gov_shard = "" // (civilian or dod) +resource_prefix = "" // Prefix used for naming and tagging resources (e.g., S3 buckets, IAM roles). diff --git a/aws-gov/tf/variables.tf b/aws-gov/tf/variables.tf new file mode 100644 index 0000000..638e1ab --- /dev/null +++ b/aws-gov/tf/variables.tf @@ -0,0 +1,66 @@ +data "aws_availability_zones" "available" { + state = "available" +} + +variable "aws_account_id" { + description = "ID of the AWS account." + type = string +} + +variable "client_id" { + description = "Client ID for authentication." + type = string + sensitive = true +} + +variable "client_secret" { + description = "Secret key for the client ID." + type = string + sensitive = true +} + +variable "databricks_account_id" { + description = "ID of the Databricks account." + type = string + sensitive = true +} + +variable "region" { + description = "Databricks only operates in AWS Gov West (us-gov-west-1)" + default = "us-gov-west-1" + validation { + condition = contains(["us-gov-west-1"], var.region) + error_message = "Valid value for var: region is (us-gov-west-1)." + } +} + +variable "region_name" { + description = "Name of the AWS region. (e.g. pendleton)" + type = map(string) + default = { + "civilian" = "pendleton" + "dod" = "pendleton-dod" + } +} + +variable "resource_prefix" { + description = "Prefix for the resource names." + type = string +} + +// AWS Gov Only Variables +variable "account_console" { + type = map(string) + default = { + "civilian" = "https://accounts.cloud.databricks.us/" + "dod" = "https://accounts-dod.cloud.databricks.us/" + } +} + +variable "databricks_gov_shard" { + description = "pick shard: civilian, dod" + validation { + condition = contains(["civilian", "dod"], var.databricks_gov_shard) + error_message = "Valid values for var: databricks_gov_shard are (civilian, dod)." + } +} \ No newline at end of file diff --git a/aws/README.md b/aws/README.md index 9900fec..dddb2b7 100644 --- a/aws/README.md +++ b/aws/README.md @@ -1,4 +1,4 @@ -# Security Reference Architecture Template +# Security Reference Architectures (SRA) - Terraform Templates ## Introduction @@ -21,7 +21,8 @@ There are four separate operation modes you can choose for the underlying networ - **Sandbox**: Sandbox or open egress. Selecting 'sandbox' as the operation mode allows traffic to flow freely to the public internet. This mode is suitable for sandbox or development scenarios where data exfiltration protection is of minimal concern, and developers need to access public APIs, packages, and more. -- **Firewall**: Firewall or limited egress. Choosing 'firewall' as the operation mode permits traffic flow only to a selected list of public addresses. This mode is applicable in situations where open internet access is necessary for certain tasks, but unfiltered traffic is not an option due to the sensitivity of the workloads or data. **NOTE**: Due to a limitation in the AWS Network Firewall's ability to use fully qualified domain names for non-HTTP/HTTPS traffic, an external data source is required for the external Hive metastore. For production scenarios, we recommend using Unity Catalog or self-hosted Hive metastores. +- **Firewall**: Firewall or limited egress. Choosing 'firewall' as the operation mode permits traffic flow only to a selected list of public addresses. This mode is applicable in situations where open internet access is necessary for certain tasks, but unfiltered traffic is not an option due to the sensitivity of the workloads or data. + - **WARNING**: Due to a limitation in AWS Network Firewall's support for fully qualified domain names (FQDNs) in non-HTTP/HTTPS traffic, an IP address is required to allow communication with the Hive Metastore. This dependency on a static IP introduces the potential for downtime if the Hive Metastore's IP changes. For sensitive production workloads, it is recommended to explore the isolated operation mode or consider alternative firewall solutions that provide better handling of dynamic IPs or FQDNs. - **Isolated**: Isolated or no egress. Opting for 'isolated' as the operation mode prevents any traffic to the public internet. Traffic is limited to AWS private endpoints, either to AWS services or the Databricks control plane. This mode should be used in cases where access to the public internet is completely unsupported. **NOTE**: Apache Derby Metastore will be required for clusters and non-serverless SQL Warehouses. For more information, please view this [knowledge article](https://kb.databricks.com/metastore/set-up-embedded-metastore). @@ -45,18 +46,10 @@ See the below networking diagrams for more information. - **Unity Catalog**: [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/index.html) is a unified governance solution for all data and AI assets including files, tables, and machine learning models. Unity Catalog provides a modern approach to granular access controls with centralized policy, auditing, and lineage tracking - all integrated into your Databricks workflow. **NOTE**: SRA creates a workspace specific catalog that is isolated to that individual workspace. To change these settings please update uc_catalog.tf under the workspace_security_modules. -## Post Workspace Deployment - -- **Service Principals**: A [Service principal](https://docs.databricks.com/administration-guide/users-groups/service-principals.html) is an identity that you create in Databricks for use with automated tools, jobs, and applications. It's against best practice to tie production workloads to individual user accounts, and so we recommend configuring these service principals within Databricks. In this template, we create an example service principal. - -- **Token Management**: [Personal access tokens](https://docs.databricks.com/dev-tools/api/latest/authentication.html) are used to access Databricks REST APIs in-lieu of passwords. In this template we create an example token and set its time-to-live. This can be set at an administrative level for all users. - -- **Secret Management** Integrating with heterogenous systems requires managing a potentially large set of credentials and safely distributing them across an organization. Instead of directly entering your credentials into a notebook, use [Databricks secrets](https://docs.databricks.com/security/secrets/index.html) to store your credentials and reference them in notebooks and jobs. In this template, we create an example secret. - - ## Optional Deployment Configurations - **Audit and Billable Usage Logs**: Databricks delivers logs to your S3 buckets. [Audit logs](https://docs.databricks.com/administration-guide/account-settings/audit-logs.html) contain two levels of events: workspace-level audit logs with workspace-level events and account-level audit logs with account-level events. In addition to these logs, you can generate additional events by enabling verbose audit logs. [Billable usage logs](https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html) are delivered daily to an AWS S3 storage bucket. There will be a separate CSV file for each workspace. This file contains historical data about the workspace's cluster usage in Databricks Units (DBUs). +- **System Tables Schemas**: System Tables provide visiblity into access, billing, compute, Lakeflow, and storage logs. These tables can be found within the system catalog in Unity Catalog. - **Cluster Example**: An example of a cluster and a cluster policy has been included. **NOTE:** Please be aware this will create a cluster within your Databricks workspace including the underlying EC2 instance. @@ -80,16 +73,11 @@ See the below networking diagrams for more information. - **Audit Log Alerting**: Audit Log Alerting, based on this [blog post](https://www.databricks.com/blog/improve-lakehouse-security-monitoring-using-system-tables-databricks-unity-catalog), creates 40+ SQL alerts to monitor for incidents based on a Zero Trust Architecture (ZTA) model. **NOTE:** Please be aware this creates a cluster, a job, and queries within your environment. -## Public Preview Features - -- **System Tables Schemas**: System Table schemas are currently in private preview. System Tables provide visiblity into access, billing, compute, and storage logs. In this deployment the metastore admin, service principle, owns the table. Additional grant statements will be needed. **NOTE:** Please note this is currently in public preview. - - ## Additional Security Recommendations and Opportunities In this section, we break down additional security recommendations and opportunities to maintain a strong security posture that either cannot be configured into this Terraform script or is very specific to individual customers (e.g. SCIM, SSO, Front-End PrivateLink, etc.) -- **Segement Workspaces for Various Levels of Data Seperation**: While Databricks has numerous capabilities for isolating different workloads, such as table ACLs and IAM passthrough for very sensitive workloads, the primary isolation method is to move sensitive workloads to a different workspace. This sometimes happens when a customer has very different teams (for example, a security team and a marketing team) who must both analyze different data in Databricks. +- **Segment Workspaces for Various Levels of Data Separation**: While Databricks has numerous capabilities for isolating different workloads, such as table ACLs and IAM passthrough for very sensitive workloads, the primary isolation method is to move sensitive workloads to a different workspace. This sometimes happens when a customer has very different teams (for example, a security team and a marketing team) who must both analyze different data in Databricks. - **Avoid Storing Production Datasets in Databricks File Store**: Because the DBFS root is accessible to all users in a workspace, all users can access any data stored here. It is important to instruct users to avoid using this location for storing sensitive data. The default location for managed tables in the Hive metastore on Databricks is the DBFS root; to prevent end users who create managed tables from writing to the DBFS root, declare a location on external storage when creating databases in the Hive metastore. @@ -109,11 +97,12 @@ In this section, we break down additional security recommendations and opportuni 3. Decide which [operation](https://github.com/databricks/terraform-databricks-sra/tree/main/aws/tf#operation-mode) mode you'd like to use. 4. Fill out `sra.tf` in place 5. Fill out `template.tfvars.example` remove the .example part of the file name -6. CD into `tf` -7. Run `terraform init` -8. Run `terraform validate` -9. From `tf` directory, run `terraform plan -var-file ../example.tfvars` -10. Run `terraform apply -var-file ../example.tfvars` +6. Configure the [AWS](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration) and [Databricks](https://registry.terraform.io/providers/databricks/databricks/latest/docs#authentication) provider authentication +7. CD into `tf` +8. Run `terraform init` +9. Run `terraform validate` +10. From `tf` directory, run `terraform plan -var-file ../example.tfvars` +11. Run `terraform apply -var-file ../example.tfvars` ## Network Diagram - Sandbox diff --git a/aws/tf/modules/sra/cmk.tf b/aws/tf/modules/sra/cmk.tf index b1f0312..dd45769 100644 --- a/aws/tf/modules/sra/cmk.tf +++ b/aws/tf/modules/sra/cmk.tf @@ -63,7 +63,8 @@ resource "aws_kms_key" "workspace_storage" { depends_on = [aws_iam_role.cross_account_role] tags = { - Resource = var.resource_prefix + Name = "${var.resource_prefix}-workspace-storage-key" + Project = var.resource_prefix } } @@ -111,7 +112,8 @@ resource "aws_kms_key" "managed_storage" { ) tags = { - Resource = var.resource_prefix + Project = var.resource_prefix + Name = "${var.resource_prefix}-managed-storage-key" } } diff --git a/aws/tf/modules/sra/credential.tf b/aws/tf/modules/sra/credential.tf index c2d7a8b..aeb6237 100644 --- a/aws/tf/modules/sra/credential.tf +++ b/aws/tf/modules/sra/credential.tf @@ -6,10 +6,11 @@ data "databricks_aws_assume_role_policy" "this" { } resource "aws_iam_role" "cross_account_role" { - name = "${var.resource_prefix}-crossaccount" + name = "${var.resource_prefix}-cross-account" assume_role_policy = data.databricks_aws_assume_role_policy.this.json tags = { - Name = "${var.resource_prefix}-crossaccount-role" + Name = "${var.resource_prefix}-cross-account" + Project = var.resource_prefix } } diff --git a/aws/tf/modules/sra/data_plane_hardening.tf b/aws/tf/modules/sra/data_plane_hardening.tf index 65655f0..6f7924b 100644 --- a/aws/tf/modules/sra/data_plane_hardening.tf +++ b/aws/tf/modules/sra/data_plane_hardening.tf @@ -8,18 +8,17 @@ module "harden_firewall" { aws = aws } - vpc_id = module.vpc[0].vpc_id - vpc_cidr_range = var.vpc_cidr_range - public_subnets_cidr = var.public_subnets_cidr - private_subnets_cidr = module.vpc[0].private_subnets_cidr_blocks - private_subnet_rt = module.vpc[0].private_route_table_ids - firewall_subnets_cidr = var.firewall_subnets_cidr - firewall_allow_list = var.firewall_allow_list - firewall_protocol_deny_list = split(",", var.firewall_protocol_deny_list) - hive_metastore_fqdn = var.hive_metastore_fqdn - availability_zones = var.availability_zones - region = var.region - resource_prefix = var.resource_prefix + vpc_id = module.vpc[0].vpc_id + vpc_cidr_range = var.vpc_cidr_range + public_subnets_cidr = var.public_subnets_cidr + private_subnets_cidr = module.vpc[0].private_subnets_cidr_blocks + private_subnet_rt = module.vpc[0].private_route_table_ids + firewall_subnets_cidr = var.firewall_subnets_cidr + firewall_allow_list = var.firewall_allow_list + hive_metastore_fqdn = var.hms_fqdn[var.region] + availability_zones = var.availability_zones + region = var.region + resource_prefix = var.resource_prefix depends_on = [module.databricks_mws_workspace] } diff --git a/aws/tf/modules/sra/data_plane_hardening/firewall/firewall.tf b/aws/tf/modules/sra/data_plane_hardening/firewall/firewall.tf index f2e6878..c32a665 100644 --- a/aws/tf/modules/sra/data_plane_hardening/firewall/firewall.tf +++ b/aws/tf/modules/sra/data_plane_hardening/firewall/firewall.tf @@ -8,7 +8,8 @@ resource "aws_subnet" "public" { availability_zone = element(var.availability_zones, count.index) map_public_ip_on_launch = true tags = { - Name = "${var.resource_prefix}-public-${element(var.availability_zones, count.index)}" + Name = "${var.resource_prefix}-public-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix } } @@ -25,7 +26,8 @@ resource "aws_nat_gateway" "ngw" { subnet_id = element(aws_subnet.public.*.id, count.index) depends_on = [aws_internet_gateway.igw] tags = { - Name = "${var.resource_prefix}-ngw-${element(var.availability_zones, count.index)}" + Name = "${var.resource_prefix}-ngw-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix } } @@ -37,13 +39,13 @@ resource "aws_route" "private" { nat_gateway_id = element(aws_nat_gateway.ngw.*.id, count.index) } - // Public RT resource "aws_route_table" "public_rt" { count = length(var.public_subnets_cidr) vpc_id = var.vpc_id tags = { - Name = "${var.resource_prefix}-public-rt-${element(var.availability_zones, count.index)}" + Name = "${var.resource_prefix}-public-rt-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix } } @@ -63,7 +65,8 @@ resource "aws_subnet" "firewall" { availability_zone = element(var.availability_zones, count.index) map_public_ip_on_launch = false tags = { - Name = "${var.resource_prefix}-firewall-${element(var.availability_zones, count.index)}" + Name = "${var.resource_prefix}-firewall-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix } } @@ -72,7 +75,8 @@ resource "aws_route_table" "firewall_rt" { count = length(var.firewall_subnets_cidr) vpc_id = var.vpc_id tags = { - Name = "${var.resource_prefix}-firewall-rt-${element(var.availability_zones, count.index)}" + Name = "${var.resource_prefix}-firewall-rt-${element(var.availability_zones, count.index)}" + Project = var.resource_prefix } } @@ -87,7 +91,8 @@ resource "aws_route_table_association" "firewall" { resource "aws_internet_gateway" "igw" { vpc_id = var.vpc_id tags = { - Name = "${var.resource_prefix}-igw" + Name = "${var.resource_prefix}-igw" + Project = var.resource_prefix } } @@ -95,7 +100,8 @@ resource "aws_internet_gateway" "igw" { resource "aws_route_table" "igw_rt" { vpc_id = var.vpc_id tags = { - Name = "${var.resource_prefix}-igw-rt" + Name = "${var.resource_prefix}-igw-rt" + Project = var.resource_prefix } } @@ -105,16 +111,34 @@ resource "aws_route_table_association" "igw" { route_table_id = aws_route_table.igw_rt.id } +// Local Map for Availability Zone to Index +locals { + az_to_index_map = { + for idx, az in var.availability_zones : + az => idx + } + + firewall_endpoints_by_az = { + for sync_state in aws_networkfirewall_firewall.nfw.firewall_status[0].sync_states : + sync_state.availability_zone => sync_state.attachment[0].endpoint_id + } + + az_to_endpoint_map = { + for az in var.availability_zones : + az => lookup(local.firewall_endpoints_by_az, az, null) + } +} + // Public Route resource "aws_route" "public" { - count = length(var.public_subnets_cidr) - route_table_id = element(aws_route_table.public_rt.*.id, count.index) + for_each = local.az_to_endpoint_map + route_table_id = aws_route_table.public_rt[local.az_to_index_map[each.key]].id destination_cidr_block = "0.0.0.0/0" - vpc_endpoint_id = tolist(aws_networkfirewall_firewall.nfw.firewall_status[0].sync_states)[count.index].attachment[0].endpoint_id + vpc_endpoint_id = each.value depends_on = [aws_networkfirewall_firewall.nfw] } -// Firewall Route +// Firewall Outbound Route resource "aws_route" "firewall_outbound" { count = length(var.firewall_subnets_cidr) route_table_id = element(aws_route_table.firewall_rt.*.id, count.index) @@ -122,12 +146,12 @@ resource "aws_route" "firewall_outbound" { gateway_id = aws_internet_gateway.igw.id } -// Add a route back to FW +// Firewall Inbound Route resource "aws_route" "firewall_inbound" { - count = length(var.public_subnets_cidr) + for_each = local.az_to_endpoint_map route_table_id = aws_route_table.igw_rt.id - destination_cidr_block = element(var.public_subnets_cidr, count.index) - vpc_endpoint_id = tolist(aws_networkfirewall_firewall.nfw.firewall_status[0].sync_states)[count.index].attachment[0].endpoint_id + destination_cidr_block = element(var.public_subnets_cidr, index(var.availability_zones, each.key)) + vpc_endpoint_id = each.value depends_on = [aws_networkfirewall_firewall.nfw] } @@ -155,22 +179,17 @@ resource "aws_networkfirewall_rule_group" "databricks_fqdn_allowlist" { } } } - } + } tags = { - Name = "${var.resource_prefix}-${var.region}-databricks-fqdn-allowlist" + Name = "${var.resource_prefix}-${var.region}-databricks-fqdn-allowlist" + Project = var.resource_prefix } } -// Data for IP allow list -data "external" "metastore_ip" { - program = ["sh", "${path.module}/metastore_ip.sh"] - - query = { - metastore_domain = var.hive_metastore_fqdn - } +data "dns_a_record_set" "metastore_dns" { + host = var.hive_metastore_fqdn } - // JDBC Firewall group IP allow list resource "aws_networkfirewall_rule_group" "databricks_metastore_allowlist" { capacity = 100 @@ -181,10 +200,28 @@ resource "aws_networkfirewall_rule_group" "databricks_metastore_allowlist" { rule_order = "STRICT_ORDER" } rules_source { + dynamic "stateful_rule" { + for_each = toset(data.dns_a_record_set.metastore_dns.addrs) + content { + action = "PASS" + header { + destination = stateful_rule.value + destination_port = 3306 + direction = "FORWARD" + protocol = "TCP" + source = "ANY" + source_port = "ANY" + } + rule_option { + keyword = "sid" + settings = ["1"] + } + } + } stateful_rule { - action = "PASS" + action = "DROP" header { - destination = data.external.metastore_ip.result["ip"] + destination = "0.0.0.0/0" destination_port = 3306 direction = "FORWARD" protocol = "TCP" @@ -193,17 +230,18 @@ resource "aws_networkfirewall_rule_group" "databricks_metastore_allowlist" { } rule_option { keyword = "sid" - settings = ["1"] + settings = ["2"] } } } } tags = { - Name = "${var.resource_prefix}-${var.region}-databricks-metastore-allowlist" + Name = "${var.resource_prefix}-${var.region}-databricks-metastore-allowlist" + Project = var.resource_prefix } } -# Firewall policy +// Firewall policy resource "aws_networkfirewall_firewall_policy" "databricks_nfw_policy" { name = "${var.resource_prefix}-firewall-policy" @@ -225,15 +263,14 @@ resource "aws_networkfirewall_firewall_policy" "databricks_nfw_policy" { priority = 2 resource_arn = aws_networkfirewall_rule_group.databricks_metastore_allowlist.arn } - } tags = { - Name = "${var.resource_prefix}-firewall-policy" + Name = "${var.resource_prefix}-firewall-policy" + Project = var.resource_prefix } } - // Firewall resource "aws_networkfirewall_firewall" "nfw" { name = "${var.resource_prefix}-nfw" @@ -246,6 +283,7 @@ resource "aws_networkfirewall_firewall" "nfw" { } } tags = { - Name = "${var.resource_prefix}-${var.region}-databricks-nfw" + Name = "${var.resource_prefix}-${var.region}-databricks-nfw" + Project = var.resource_prefix } } \ No newline at end of file diff --git a/aws/tf/modules/sra/data_plane_hardening/firewall/metastore_ip.sh b/aws/tf/modules/sra/data_plane_hardening/firewall/metastore_ip.sh deleted file mode 100755 index 6062790..0000000 --- a/aws/tf/modules/sra/data_plane_hardening/firewall/metastore_ip.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -eval "$(jq -r '@sh "METASTORE_DOMAIN=\(.metastore_domain)"')" - -IP=$(dig +short $METASTORE_DOMAIN | tail -n1) -echo "Resolved IP: $IP" >&2 - -if [ -z "$IP" ]; then - echo "Error: Failed to resolve IP for $METASTORE_DOMAIN" >&2 - exit 1 -fi - -jq -n --arg ip "$IP" '{"ip":$ip}' \ No newline at end of file diff --git a/aws/tf/modules/sra/data_plane_hardening/firewall/provider.tf b/aws/tf/modules/sra/data_plane_hardening/firewall/provider.tf index 7afdcf4..7617f6b 100644 --- a/aws/tf/modules/sra/data_plane_hardening/firewall/provider.tf +++ b/aws/tf/modules/sra/data_plane_hardening/firewall/provider.tf @@ -3,5 +3,8 @@ terraform { aws = { source = "hashicorp/aws" } + dns = { + source = "hashicorp/dns" + } } } \ No newline at end of file diff --git a/aws/tf/modules/sra/data_plane_hardening/firewall/variables.tf b/aws/tf/modules/sra/data_plane_hardening/firewall/variables.tf index 4bb8ed9..852e484 100644 --- a/aws/tf/modules/sra/data_plane_hardening/firewall/variables.tf +++ b/aws/tf/modules/sra/data_plane_hardening/firewall/variables.tf @@ -1,47 +1,43 @@ -variable "vpc_id" { - type = string +variable "availability_zones" { + type = list(string) } -variable "vpc_cidr_range" { - type = string +variable "firewall_allow_list" { + type = list(string) } -variable "public_subnets_cidr" { +variable "firewall_subnets_cidr" { type = list(string) } -variable "private_subnets_cidr" { - type = list(string) +variable "hive_metastore_fqdn" { + type = string } variable "private_subnet_rt" { type = list(string) } -variable "firewall_subnets_cidr" { +variable "private_subnets_cidr" { type = list(string) } -variable "firewall_allow_list" { +variable "public_subnets_cidr" { type = list(string) } -variable "hive_metastore_fqdn" { +variable "region" { type = string } -variable "availability_zones" { - type = list(string) +variable "resource_prefix" { + type = string } -variable "region" { +variable "vpc_cidr_range" { type = string } -variable "resource_prefix" { +variable "vpc_id" { type = string } - -variable "firewall_protocol_deny_list" { - type = list(string) -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account.tf b/aws/tf/modules/sra/databricks_account.tf index 924b1ca..9469d2e 100644 --- a/aws/tf/modules/sra/databricks_account.tf +++ b/aws/tf/modules/sra/databricks_account.tf @@ -15,7 +15,6 @@ module "log_delivery" { // Create Unity Catalog Metastore - No Root Storage module "uc_init" { - count = var.metastore_exists == false ? 1 : 0 source = "./databricks_account/uc_init" providers = { databricks = databricks.mws @@ -26,6 +25,7 @@ module "uc_init" { resource_prefix = var.resource_prefix region = var.region metastore_name = join("", [var.resource_prefix, "-", var.region, "-", "uc"]) + metastore_exists = var.metastore_exists } // Unity Catalog Assignment @@ -35,12 +35,10 @@ module "uc_assignment" { databricks = databricks.mws } - metastore_id = var.metastore_exists ? null : module.uc_init[0].metastore_id + metastore_id = module.uc_init.metastore_id region = var.region workspace_id = module.databricks_mws_workspace.workspace_id - depends_on = [ - module.databricks_mws_workspace - ] + depends_on = [module.databricks_mws_workspace, module.uc_init] } // Create Databricks Workspace @@ -66,22 +64,6 @@ module "databricks_mws_workspace" { workspace_storage_key_alias = aws_kms_alias.workspace_storage_key_alias.name } -// Service Principal -module "service_principal" { - source = "./databricks_account/service_principal" - providers = { - databricks = databricks.mws - } - - created_workspace_id = module.databricks_mws_workspace.workspace_id - workspace_admin_service_principal_name = var.workspace_admin_service_principal_name - - depends_on = [ - module.databricks_mws_workspace, - module.uc_assignment - ] -} - // User Workspace Assignment (Admin) module "user_assignment" { source = "./databricks_account/user_assignment" @@ -91,9 +73,5 @@ module "user_assignment" { created_workspace_id = module.databricks_mws_workspace.workspace_id workspace_access = var.user_workspace_admin - - depends_on = [ - module.databricks_mws_workspace, - module.uc_assignment - ] + depends_on = [module.uc_assignment, module.databricks_mws_workspace] } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf b/aws/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf index 4c6b391..9ad0a67 100644 --- a/aws/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf +++ b/aws/tf/modules/sra/databricks_account/logging_configuration/logging_configuration.tf @@ -5,7 +5,8 @@ resource "aws_s3_bucket" "log_delivery" { bucket = "${var.resource_prefix}-log-delivery" force_destroy = true tags = { - Name = "${var.resource_prefix}-log-delivery" + Name = "${var.resource_prefix}-log-delivery" + Project = var.resource_prefix } } @@ -96,7 +97,8 @@ resource "aws_iam_role" "log_delivery" { description = "(${var.resource_prefix}) Log Delivery Role" assume_role_policy = data.databricks_aws_assume_role_policy.log_delivery.json tags = { - Name = "${var.resource_prefix}-log-delivery-role" + Name = "${var.resource_prefix}-log-delivery-role" + Project = var.resource_prefix } } diff --git a/aws/tf/modules/sra/databricks_account/logging_configuration/variables.tf b/aws/tf/modules/sra/databricks_account/logging_configuration/variables.tf index 55aaac6..53cfbc5 100644 --- a/aws/tf/modules/sra/databricks_account/logging_configuration/variables.tf +++ b/aws/tf/modules/sra/databricks_account/logging_configuration/variables.tf @@ -1,7 +1,7 @@ -variable "resource_prefix" { +variable "databricks_account_id" { type = string } -variable "databricks_account_id" { +variable "resource_prefix" { type = string } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/service_principal/output.tf b/aws/tf/modules/sra/databricks_account/service_principal/output.tf deleted file mode 100644 index 678c54b..0000000 --- a/aws/tf/modules/sra/databricks_account/service_principal/output.tf +++ /dev/null @@ -1,3 +0,0 @@ -output "service_principal_id" { - value = databricks_service_principal.sp.id -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/service_principal/service_principal.tf b/aws/tf/modules/sra/databricks_account/service_principal/service_principal.tf deleted file mode 100644 index a7d25d5..0000000 --- a/aws/tf/modules/sra/databricks_account/service_principal/service_principal.tf +++ /dev/null @@ -1,12 +0,0 @@ -// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/service_principal - -resource "databricks_service_principal" "sp" { - display_name = var.workspace_admin_service_principal_name - allow_cluster_create = true -} - -resource "databricks_mws_permission_assignment" "admin_sp" { - workspace_id = var.created_workspace_id - principal_id = databricks_service_principal.sp.id - permissions = ["ADMIN"] -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/service_principal/variables.tf b/aws/tf/modules/sra/databricks_account/service_principal/variables.tf deleted file mode 100644 index 118a72b..0000000 --- a/aws/tf/modules/sra/databricks_account/service_principal/variables.tf +++ /dev/null @@ -1,8 +0,0 @@ -variable "created_workspace_id" { - type = string -} - -variable "workspace_admin_service_principal_name" { - description = "Service principal name" - type = string -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf b/aws/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf index bae1aa0..5ead29d 100644 --- a/aws/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf +++ b/aws/tf/modules/sra/databricks_account/uc_assignment/uc_assignment.tf @@ -1,11 +1,6 @@ // Metastore Assignment -data "databricks_metastore" "this" { - region = var.region -} - resource "databricks_metastore_assignment" "default_metastore" { workspace_id = var.workspace_id - metastore_id = var.metastore_id == null ? data.databricks_metastore.this.id : var.metastore_id - default_catalog_name = "hive_metastore" + metastore_id = var.metastore_id } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/uc_assignment/variables.tf b/aws/tf/modules/sra/databricks_account/uc_assignment/variables.tf index b97ffde..8c922ed 100644 --- a/aws/tf/modules/sra/databricks_account/uc_assignment/variables.tf +++ b/aws/tf/modules/sra/databricks_account/uc_assignment/variables.tf @@ -2,10 +2,10 @@ variable "metastore_id" { type = string } -variable "workspace_id" { +variable "region" { type = string } -variable "region" { +variable "workspace_id" { type = string } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/uc_init/outputs.tf b/aws/tf/modules/sra/databricks_account/uc_init/outputs.tf index 6f7a596..c122a8c 100644 --- a/aws/tf/modules/sra/databricks_account/uc_init/outputs.tf +++ b/aws/tf/modules/sra/databricks_account/uc_init/outputs.tf @@ -1,3 +1,3 @@ output "metastore_id" { - value = databricks_metastore.this.id + value = var.metastore_exists ? data.databricks_metastore.this[0].id : databricks_metastore.this[0].id } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/uc_init/uc_init.tf b/aws/tf/modules/sra/databricks_account/uc_init/uc_init.tf index a5d1102..34df58d 100644 --- a/aws/tf/modules/sra/databricks_account/uc_init/uc_init.tf +++ b/aws/tf/modules/sra/databricks_account/uc_init/uc_init.tf @@ -1,7 +1,13 @@ // Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog -// Metastore +// Optional data source - only run if the metastore exists +data "databricks_metastore" "this" { + count = var.metastore_exists ? 1 : 0 + region = var.region +} + resource "databricks_metastore" "this" { + count = var.metastore_exists ? 0 : 1 name = "${var.resource_prefix}-${var.region}-unity-catalog" region = var.region force_destroy = true diff --git a/aws/tf/modules/sra/databricks_account/uc_init/variables.tf b/aws/tf/modules/sra/databricks_account/uc_init/variables.tf index 514f460..ec1a35d 100644 --- a/aws/tf/modules/sra/databricks_account/uc_init/variables.tf +++ b/aws/tf/modules/sra/databricks_account/uc_init/variables.tf @@ -2,11 +2,11 @@ variable "aws_account_id" { type = string } -variable "resource_prefix" { +variable "databricks_account_id" { type = string } -variable "databricks_account_id" { +variable "metastore_exists" { type = string } @@ -16,4 +16,8 @@ variable "metastore_name" { variable "region" { type = string +} + +variable "resource_prefix" { + type = string } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_account/workspace/variables.tf b/aws/tf/modules/sra/databricks_account/workspace/variables.tf index cf1cc18..07748d4 100644 --- a/aws/tf/modules/sra/databricks_account/workspace/variables.tf +++ b/aws/tf/modules/sra/databricks_account/workspace/variables.tf @@ -1,52 +1,52 @@ -variable "bucket_name" { +variable "backend_relay" { type = string } -variable "cross_account_role_arn" { +variable "backend_rest" { type = string } -variable "databricks_account_id" { +variable "bucket_name" { type = string } -variable "resource_prefix" { +variable "cross_account_role_arn" { type = string } -variable "region" { +variable "databricks_account_id" { type = string } -variable "security_group_ids" { - type = list(string) +variable "managed_storage_key" { + type = string } -variable "subnet_ids" { - type = list(string) +variable "managed_storage_key_alias" { + type = string } -variable "vpc_id" { +variable "region" { type = string } -variable "backend_rest" { +variable "resource_prefix" { type = string } -variable "backend_relay" { - type = string +variable "security_group_ids" { + type = list(string) } -variable "managed_storage_key" { - type = string +variable "subnet_ids" { + type = list(string) } -variable "workspace_storage_key" { +variable "vpc_id" { type = string } -variable "managed_storage_key_alias" { +variable "workspace_storage_key" { type = string } diff --git a/aws/tf/modules/sra/databricks_workspace.tf b/aws/tf/modules/sra/databricks_workspace.tf index 531354b..f25d0af 100644 --- a/aws/tf/modules/sra/databricks_workspace.tf +++ b/aws/tf/modules/sra/databricks_workspace.tf @@ -7,16 +7,15 @@ module "uc_catalog" { databricks = databricks.created_workspace } - databricks_account_id = var.databricks_account_id - aws_account_id = var.aws_account_id - resource_prefix = var.resource_prefix - uc_catalog_name = "${var.resource_prefix}-catalog-${module.databricks_mws_workspace.workspace_id}" - workspace_id = module.databricks_mws_workspace.workspace_id - workspace_catalog_admin = var.workspace_catalog_admin - - depends_on = [ - module.databricks_mws_workspace, module.uc_assignment - ] + databricks_account_id = var.databricks_account_id + aws_account_id = var.aws_account_id + resource_prefix = var.resource_prefix + uc_catalog_name = "${var.resource_prefix}-catalog-${module.databricks_mws_workspace.workspace_id}" + cmk_admin_arn = var.cmk_admin_arn == null ? "arn:aws:iam::${var.aws_account_id}:root" : var.cmk_admin_arn + workspace_id = module.databricks_mws_workspace.workspace_id + user_workspace_catalog_admin = var.user_workspace_catalog_admin + + depends_on = [module.databricks_mws_workspace, module.uc_assignment] } // Create Read-Only Storage Location for Data Bucket & External Location @@ -32,10 +31,6 @@ module "uc_external_location" { resource_prefix = var.resource_prefix read_only_data_bucket = var.read_only_data_bucket read_only_external_location_admin = var.read_only_external_location_admin - - depends_on = [ - module.databricks_mws_workspace, module.uc_assignment - ] } // Workspace Admin Configuration @@ -45,34 +40,6 @@ module "admin_configuration" { providers = { databricks = databricks.created_workspace } - - depends_on = [ - module.databricks_mws_workspace - ] -} - -// Token Management -module "token_management" { - source = "./databricks_workspace/workspace_security_modules/token_management" - providers = { - databricks = databricks.created_workspace - } - - depends_on = [ - module.databricks_mws_workspace - ] -} - -// Secret Management -module "secret_management" { - source = "./databricks_workspace/workspace_security_modules/secret_management" - providers = { - databricks = databricks.created_workspace - } - - depends_on = [ - module.databricks_mws_workspace - ] } // IP Access Lists - Optional @@ -84,10 +51,6 @@ module "ip_access_list" { } ip_addresses = var.ip_addresses - - depends_on = [ - module.databricks_mws_workspace - ] } // Create Create Cluster - Optional @@ -99,24 +62,18 @@ module "cluster_configuration" { } compliance_security_profile_egress_ports = var.compliance_security_profile_egress_ports - secret_config_reference = module.secret_management.config_reference resource_prefix = var.resource_prefix - depends_on = [ - module.databricks_mws_workspace, module.secret_management - ] + operation_mode = var.operation_mode } -// Public Preview - System Table Schemas - Optional -module "public_preview_system_table" { - source = "./databricks_workspace/public_preview/system_schema/" +// System Table Schemas Enablement - Optional +module "system_table" { + source = "./databricks_workspace/workspace_security_modules/system_schema/" count = var.enable_system_tables_schema_boolean ? 1 : 0 providers = { databricks = databricks.created_workspace } - - depends_on = [ - module.databricks_mws_workspace - ] + depends_on = [ module.uc_assignment ] } // SAT Implementation - Optional @@ -127,16 +84,18 @@ module "security_analysis_tool" { databricks = databricks.created_workspace } - databricks_url = module.databricks_mws_workspace.workspace_url - workspace_PAT = module.service_principal.service_principal_id - workspace_id = module.databricks_mws_workspace.workspace_id - account_console_id = var.databricks_account_id - client_id = var.client_id - client_secret = var.client_secret - use_sp_auth = true + databricks_url = module.databricks_mws_workspace.workspace_url + workspace_id = module.databricks_mws_workspace.workspace_id + account_console_id = var.databricks_account_id + client_id = var.client_id + client_secret = var.client_secret + use_sp_auth = true + proxies = {} + analysis_schema_name = "SAT" + depends_on = [ - module.databricks_mws_workspace, module.service_principal + module.databricks_mws_workspace ] } @@ -149,8 +108,4 @@ module "audit_log_alerting" { } alert_emails = [var.user_workspace_admin] - - depends_on = [ - module.databricks_mws_workspace, module.uc_assignment - ] } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf index a683b8d..b055acf 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/provider.tf @@ -7,8 +7,10 @@ terraform { } module "common" { - source = "../common/" - account_console_id = var.account_console_id - workspace_id = var.workspace_id - sqlw_id = var.sqlw_id + source = "../common/" + account_console_id = var.account_console_id + workspace_id = var.workspace_id + sqlw_id = var.sqlw_id + analysis_schema_name = var.analysis_schema_name + proxies = var.proxies } diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf index 21a0178..db695c4 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/secrets.tf @@ -1,5 +1,17 @@ ### AWS Specific Secrets +resource "databricks_secret" "user" { + key = "user" + string_value = var.account_user + scope = module.common.secret_scope_id +} + +resource "databricks_secret" "pass" { + key = "pass" + string_value = var.account_pass + scope = module.common.secret_scope_id +} + resource "databricks_secret" "use_sp_auth" { key = "use-sp-auth" string_value = var.use_sp_auth diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf index a3cccad..84420a8 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/aws/variables.tf @@ -8,11 +8,6 @@ variable "workspace_id" { type = string } -variable "workspace_PAT" { - description = "PAT should look like dapixxxxxxxxxxxxxxxxxxxx" - type = string -} - variable "account_console_id" { description = "Databricks Account Console ID" type = string @@ -30,10 +25,22 @@ variable "sqlw_id" { ### AWS Specific Variables +variable "account_user" { + description = "Account Console Username" + type = string + default = " " +} + +variable "account_pass" { + description = "Account Console Password" + type = string + default = " " +} + variable "use_sp_auth" { description = "Authenticate with Service Principal OAuth tokens instead of user and password" type = bool - default = false + default = true } variable "client_id" { @@ -47,3 +54,13 @@ variable "client_secret" { type = string default = "value" } + +variable "analysis_schema_name" { + type = string + description = "Name of the schema to be used for analysis" +} + +variable "proxies" { + type = map(any) + description = "Proxies to be used for Databricks API calls" +} diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf index 047a810..7e46fa1 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/jobs.tf @@ -1,53 +1,66 @@ resource "databricks_job" "initializer" { name = "SAT Initializer Notebook (one-time)" - new_cluster { - num_workers = 5 - spark_version = data.databricks_spark_version.latest_lts.id - node_type_id = data.databricks_node_type.smallest.id - runtime_engine = "PHOTON" - dynamic "gcp_attributes" { - for_each = var.gcp_impersonate_service_account == "" ? [] : [var.gcp_impersonate_service_account] - content { - google_service_account = var.gcp_impersonate_service_account + job_cluster { + job_cluster_key = "job_cluster" + new_cluster { + num_workers = 5 + spark_version = data.databricks_spark_version.latest_lts.id + node_type_id = data.databricks_node_type.smallest.id + runtime_engine = "PHOTON" + dynamic "gcp_attributes" { + for_each = var.gcp_impersonate_service_account == "" ? [] : [var.gcp_impersonate_service_account] + content { + google_service_account = var.gcp_impersonate_service_account + } } } } - library { - pypi { - package = "dbl-sat-sdk" + task { + task_key = "Initializer" + job_cluster_key = "job_cluster" + library { + pypi { + package = "dbl-sat-sdk" + } + } + notebook_task { + notebook_path = "${databricks_repo.security_analysis_tool.workspace_path}/notebooks/security_analysis_initializer" } - } - - notebook_task { - notebook_path = "${databricks_repo.security_analysis_tool.path}/notebooks/security_analysis_initializer" } } resource "databricks_job" "driver" { name = "SAT Driver Notebook" - new_cluster { - num_workers = 5 - spark_version = data.databricks_spark_version.latest_lts.id - node_type_id = data.databricks_node_type.smallest.id - runtime_engine = "PHOTON" - dynamic "gcp_attributes" { - for_each = var.gcp_impersonate_service_account == "" ? [] : [var.gcp_impersonate_service_account] - content { - google_service_account = var.gcp_impersonate_service_account + job_cluster { + job_cluster_key = "job_cluster" + new_cluster { + num_workers = 5 + spark_version = data.databricks_spark_version.latest_lts.id + node_type_id = data.databricks_node_type.smallest.id + runtime_engine = "PHOTON" + dynamic "gcp_attributes" { + for_each = var.gcp_impersonate_service_account == "" ? [] : [var.gcp_impersonate_service_account] + content { + google_service_account = var.gcp_impersonate_service_account + } } } } - library { - pypi { - package = "dbl-sat-sdk" - } - } - notebook_task { - notebook_path = "${databricks_repo.security_analysis_tool.path}/notebooks/security_analysis_driver" + task { + task_key = "Driver" + job_cluster_key = "job_cluster" + library { + pypi { + package = "dbl-sat-sdk" + } + } + notebook_task { + notebook_path = "${databricks_repo.security_analysis_tool.workspace_path}/notebooks/security_analysis_driver" + } } schedule { diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf index 1d847d2..e5c4d7f 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/provider.tf @@ -4,4 +4,5 @@ terraform { source = "databricks/databricks" } } -} \ No newline at end of file +} + diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf index 87c1dc5..7b21149 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/repo.tf @@ -1,5 +1,7 @@ #Make sure Files in Repos option is enabled in Workspace Admin Console > Workspace Settings resource "databricks_repo" "security_analysis_tool" { - url = "https://github.com/databricks-industry-solutions/security-analysis-tool.git" + url = "https://github.com/databricks-industry-solutions/security-analysis-tool.git" + branch = "main" + path = "/Workspace/Applications/SAT_TF" } diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf index 40ef89d..30a35f6 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/secrets.tf @@ -8,17 +8,6 @@ resource "databricks_secret" "user_email" { scope = databricks_secret_scope.sat.id } -resource "databricks_token" "pat" { - lifetime_seconds = 86400 * 365 - comment = "Security Analysis Tool" -} - -resource "databricks_secret" "pat" { - key = "sat-token-${var.workspace_id}" - string_value = databricks_token.pat.token_value - scope = databricks_secret_scope.sat.id -} - resource "databricks_secret" "account_console_id" { key = "account-console-id" string_value = var.account_console_id @@ -30,3 +19,16 @@ resource "databricks_secret" "sql_warehouse_id" { string_value = var.sqlw_id == "new" ? databricks_sql_endpoint.new[0].id : data.databricks_sql_warehouse.old[0].id scope = databricks_secret_scope.sat.id } + +resource "databricks_secret" "analysis_schema_name" { + key = "analysis_schema_name" + string_value = var.analysis_schema_name + scope = databricks_secret_scope.sat.id +} + +resource "databricks_secret" "proxies" { + key = "proxies" + string_value = jsonencode(var.proxies) + scope = databricks_secret_scope.sat.id +} + diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf index 9a99283..150ac5a 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf @@ -34,3 +34,13 @@ variable "gcp_impersonate_service_account" { description = "GCP Service Account to impersonate (e.g. xyz-sa-2@project.iam.gserviceaccount.com)" default = "" } + +variable "analysis_schema_name" { + type = string + description = "Name of the schema to be used for analysis" +} + +variable "proxies" { + type = map(any) + description = "Proxies to be used for Databricks API calls" +} diff --git a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf index 046e2d2..f684f88 100644 --- a/aws/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf +++ b/aws/tf/modules/sra/databricks_workspace/solution_accelerators/system_tables_audit_log/variables.tf @@ -1,10 +1,10 @@ +variable "alert_emails" { + type = list(string) + description = "List of emails to notify when alerts are fired" +} + variable "warehouse_id" { type = string default = "" description = "Optional Warehouse ID to run queries on. If not provided, new SQL Warehouse is created" -} - -variable "alert_emails" { - type = list(string) - description = "List of emails to notify when alerts are fired" } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf index 87b71b9..ed3d319 100644 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/cluster_configuration.tf @@ -14,39 +14,65 @@ locals { }, "autotermination_minutes" : { "type" : "fixed", - "value" : 60, + "value" : 10, "hidden" : true }, - "custom_tags.Example" : { + "custom_tags.Project" : { "type" : "fixed", "value" : var.resource_prefix - } + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionURL" : null, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionDriverName" : null, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionUserName" : null, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionPassword" : null } + + isolated_policy = merge( + local.default_policy, + { + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionURL" : { + "type" : "fixed", + "value" : "jdbc:derby:memory:myInMemDB;create=true" + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionDriverName" : { + "type" : "fixed", + "value" : "org.apache.derby.jdbc.EmbeddedDriver" + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionUserName" : { + "type" : "fixed", + "value" : "" + }, + "spark_conf.spark.hadoop.javax.jdo.option.ConnectionPassword" : { + "type" : "fixed", + "value" : "" + } + } + ) + + selected_policy = var.operation_mode == "isolated" ? local.isolated_policy : local.default_policy + + final_policy = { for k, v in local.selected_policy : k => v if v != null } } resource "databricks_cluster_policy" "example" { name = "Example Cluster Policy" - definition = jsonencode(local.default_policy) + definition = jsonencode(local.final_policy) } // Cluster Creation resource "databricks_cluster" "example" { - cluster_name = "Shared Cluster" - data_security_mode = "USER_ISOLATION" - spark_version = data.databricks_spark_version.latest_lts.id - node_type_id = var.compliance_security_profile_egress_ports ? "i3en.xlarge" : "i3.xlarge" - policy_id = databricks_cluster_policy.example.id + cluster_name = "Shared Cluster" + data_security_mode = "USER_ISOLATION" + spark_version = data.databricks_spark_version.latest_lts.id + node_type_id = var.compliance_security_profile_egress_ports ? "i3en.xlarge" : "i3.xlarge" + policy_id = databricks_cluster_policy.example.id + autotermination_minutes = 10 autoscale { min_workers = 1 max_workers = 2 } - spark_conf = { - # Add additional spark configurations here - "secret.example" = var.secret_config_reference - } - depends_on = [ databricks_cluster_policy.example ] diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf index 615c442..69f9a6a 100644 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/cluster_configuration/variables.tf @@ -1,13 +1,12 @@ -variable "resource_prefix" { - type = string +variable "compliance_security_profile_egress_ports" { + type = bool + nullable = false } -variable "secret_config_reference" { +variable "operation_mode" { type = string } -variable "compliance_security_profile_egress_ports" { - type = bool - description = "Add 2443 to security group configuration or nitro instance" - nullable = false +variable "resource_prefix" { + type = string } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/output.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/output.tf deleted file mode 100644 index ca80979..0000000 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/output.tf +++ /dev/null @@ -1,3 +0,0 @@ -output "config_reference" { - value = databricks_secret.example_app_secret.config_reference -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/secret_management.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/secret_management.tf deleted file mode 100644 index 65f0be9..0000000 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/secret_management/secret_management.tf +++ /dev/null @@ -1,11 +0,0 @@ -// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret - -resource "databricks_secret_scope" "app" { - name = "application-secret-scope" -} - -resource "databricks_secret" "example_app_secret" { - key = "example_api_secret" - string_value = "value that should be hidden from Terraform!" - scope = databricks_secret_scope.app.id -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/provider.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/provider.tf new file mode 100644 index 0000000..1d847d2 --- /dev/null +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/provider.tf @@ -0,0 +1,7 @@ +terraform { + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/system_schema.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/system_schema.tf new file mode 100644 index 0000000..180bf9f --- /dev/null +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/system_schema/system_schema.tf @@ -0,0 +1,25 @@ +// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/system_schema + +resource "databricks_system_schema" "access" { + schema = "access" +} + +resource "databricks_system_schema" "billing" { + schema = "billing" +} + +resource "databricks_system_schema" "compute" { + schema = "compute" +} + +resource "databricks_system_schema" "lakeflow" { + schema = "lakeflow" +} + +resource "databricks_system_schema" "marketplace" { + schema = "marketplace" +} + +resource "databricks_system_schema" "storage" { + schema = "storage" +} diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/token_management/token_management.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/token_management/token_management.tf deleted file mode 100644 index 980ab4e..0000000 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/token_management/token_management.tf +++ /dev/null @@ -1,7 +0,0 @@ -// Terraform Documentation: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/token - -resource "databricks_token" "pat" { - comment = "Terraform Provisioning" - // 30 day token - lifetime_seconds = 2592000 -} \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf index 69249c1..6595b61 100644 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/uc_catalog.tf @@ -1,90 +1,82 @@ resource "null_resource" "previous" {} resource "time_sleep" "wait_30_seconds" { - depends_on = [null_resource.previous] - + depends_on = [null_resource.previous] create_duration = "30s" } - -// Unity Catalog Trust Policy -data "aws_iam_policy_document" "passrole_for_unity_catalog_catalog" { - statement { - effect = "Allow" - actions = ["sts:AssumeRole"] - principals { - identifiers = ["arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"] - type = "AWS" - } - condition { - test = "StringEquals" - variable = "sts:ExternalId" - values = [var.databricks_account_id] - } - } - statement { - sid = "ExplicitSelfRoleAssumption" - effect = "Allow" - actions = ["sts:AssumeRole"] - principals { - type = "AWS" - identifiers = ["arn:aws:iam::${var.aws_account_id}:root"] - } - condition { - test = "ArnLike" - variable = "aws:PrincipalArn" - values = ["arn:aws:iam::${var.aws_account_id}:role/${var.resource_prefix}-unity-catalog-${var.workspace_id}"] - } - condition { - test = "StringEquals" - variable = "sts:ExternalId" - values = [var.databricks_account_id] - } - } +// Unity Catalog Trust Policy - Data Source +data "databricks_aws_unity_catalog_assume_role_policy" "unity_catalog" { + aws_account_id = var.aws_account_id + role_name = "${var.resource_prefix}-catalog-${var.workspace_id}" + external_id = var.databricks_account_id } // Unity Catalog Role resource "aws_iam_role" "unity_catalog_role" { - name = "${var.resource_prefix}-unity-catalog-${var.workspace_id}" - assume_role_policy = data.aws_iam_policy_document.passrole_for_unity_catalog_catalog.json + name = "${var.resource_prefix}-catalog-${var.workspace_id}" + assume_role_policy = data.databricks_aws_unity_catalog_assume_role_policy.unity_catalog.json tags = { - Name = "${var.resource_prefix}-unity-catalog" + Name = "${var.resource_prefix}-catalog-${var.workspace_id}" + Project = var.resource_prefix } } -// Unity Catalog IAM Policy -data "aws_iam_policy_document" "unity_catalog_iam_policy" { - statement { - actions = [ - "s3:GetObject", - "s3:GetObjectVersion", - "s3:PutObject", - "s3:PutObjectAcl", - "s3:DeleteObject", - "s3:ListBucket", - "s3:GetBucketLocation" - ] - - resources = [ - "arn:aws:s3:::${var.uc_catalog_name}/*", - "arn:aws:s3:::${var.uc_catalog_name}" - ] - - effect = "Allow" - } - - statement { - actions = ["sts:AssumeRole"] - resources = ["arn:aws:iam::${var.aws_account_id}:role/${var.resource_prefix}-unity-catalog-${var.workspace_id}"] - effect = "Allow" - } +// Unity Catalog Policy - Data Source +data "databricks_aws_unity_catalog_policy" "unity_catalog_iam_policy" { + aws_account_id = var.aws_account_id + bucket_name = var.uc_catalog_name + role_name = "${var.resource_prefix}-catalog-${var.workspace_id}" + kms_name = aws_kms_alias.catalog_storage_key_alias.arn } // Unity Catalog Policy resource "aws_iam_role_policy" "unity_catalog" { - name = "${var.resource_prefix}-unity-catalog-policy-${var.workspace_id}" + name = "${var.resource_prefix}-catalog-policy-${var.workspace_id}" role = aws_iam_role.unity_catalog_role.id - policy = data.aws_iam_policy_document.unity_catalog_iam_policy.json + policy = data.databricks_aws_unity_catalog_policy.unity_catalog_iam_policy.json +} + +// Unity Catalog KMS +resource "aws_kms_key" "catalog_storage" { + description = "KMS key for Databricks catalog storage ${var.workspace_id}" + policy = jsonencode({ + Version : "2012-10-17", + "Id" : "key-policy-catalog-storage-${var.workspace_id}", + Statement : [ + { + "Sid" : "Enable IAM User Permissions", + "Effect" : "Allow", + "Principal" : { + "AWS" : [var.cmk_admin_arn] + }, + "Action" : "kms:*", + "Resource" : "*" + }, + { + "Sid" : "Allow IAM Role to use the key", + "Effect" : "Allow", + "Principal" : { + "AWS" : "arn:aws:iam::${var.aws_account_id}:role/${var.resource_prefix}-catalog-${var.workspace_id}" + }, + "Action" : [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey*" + ], + "Resource" : "*" + } + ] + }) + tags = { + Name = "${var.resource_prefix}-catalog-storage-${var.workspace_id}-key" + Project = var.resource_prefix + } +} + +resource "aws_kms_alias" "catalog_storage_key_alias" { + name = "alias/${var.resource_prefix}-catalog-storage-${var.workspace_id}-key" + target_key_id = aws_kms_key.catalog_storage.id } @@ -93,7 +85,8 @@ resource "aws_s3_bucket" "unity_catalog_bucket" { bucket = var.uc_catalog_name force_destroy = true tags = { - Name = var.uc_catalog_name + Name = var.uc_catalog_name + Project = var.resource_prefix } } @@ -106,12 +99,14 @@ resource "aws_s3_bucket_versioning" "unity_catalog_versioning" { resource "aws_s3_bucket_server_side_encryption_configuration" "unity_catalog" { bucket = aws_s3_bucket.unity_catalog_bucket.bucket - rule { + bucket_key_enabled = true apply_server_side_encryption_by_default { - sse_algorithm = "AES256" + sse_algorithm = "aws:kms" + kms_master_key_id = aws_kms_key.catalog_storage.arn } } + depends_on = [aws_kms_alias.catalog_storage_key_alias] } resource "aws_s3_bucket_public_access_block" "unity_catalog" { @@ -129,7 +124,8 @@ resource "databricks_storage_credential" "workspace_catalog_storage_credential" aws_iam_role { role_arn = aws_iam_role.unity_catalog_role.arn } - depends_on = [aws_iam_role.unity_catalog_role, time_sleep.wait_30_seconds] + depends_on = [aws_iam_role.unity_catalog_role, time_sleep.wait_30_seconds] + isolation_mode = "ISOLATION_MODE_ISOLATED" } // External Location @@ -137,15 +133,13 @@ resource "databricks_external_location" "workspace_catalog_external_location" { name = var.uc_catalog_name url = "s3://${var.uc_catalog_name}/catalog/" credential_name = databricks_storage_credential.workspace_catalog_storage_credential.id - skip_validation = true - read_only = false - comment = "Managed by TF" + comment = "External location for catalog ${var.uc_catalog_name}" + isolation_mode = "ISOLATION_MODE_ISOLATED" } - // Workspace Catalog resource "databricks_catalog" "workspace_catalog" { - name = var.uc_catalog_name + name = replace(var.uc_catalog_name, "-", "_") comment = "This catalog is for workspace - ${var.workspace_id}" isolation_mode = "ISOLATED" storage_root = "s3://${var.uc_catalog_name}/catalog/" @@ -155,10 +149,17 @@ resource "databricks_catalog" "workspace_catalog" { depends_on = [databricks_external_location.workspace_catalog_external_location] } +// Set Workspace Catalog as Default +resource "databricks_default_namespace_setting" "this" { + namespace { + value = replace(var.uc_catalog_name, "-", "_") + } +} + // Grant Admin Catalog Perms resource "databricks_grant" "workspace_catalog" { catalog = databricks_catalog.workspace_catalog.name - principal = var.workspace_catalog_admin + principal = var.user_workspace_catalog_admin privileges = ["ALL_PRIVILEGES"] } diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf index 6420927..b029f1e 100644 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_catalog/variables.tf @@ -2,7 +2,7 @@ variable "aws_account_id" { type = string } -variable "resource_prefix" { +variable "cmk_admin_arn" { type = string } @@ -10,7 +10,7 @@ variable "databricks_account_id" { type = string } -variable "workspace_id" { +variable "resource_prefix" { type = string } @@ -18,6 +18,10 @@ variable "uc_catalog_name" { type = string } -variable "workspace_catalog_admin" { +variable "user_workspace_catalog_admin" { + type = string +} + +variable "workspace_id" { type = string } \ No newline at end of file diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf index 54fc08c..735357b 100644 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/uc_external_location.tf @@ -7,54 +7,25 @@ resource "time_sleep" "wait_30_seconds" { } // Storage Credential Trust Policy -data "aws_iam_policy_document" "passrole_for_storage_credential" { - statement { - effect = "Allow" - actions = ["sts:AssumeRole"] - principals { - identifiers = ["arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"] - type = "AWS" - } - condition { - test = "StringEquals" - variable = "sts:ExternalId" - values = [var.databricks_account_id] - } - } - statement { - sid = "ExplicitSelfRoleAssumption" - effect = "Allow" - actions = ["sts:AssumeRole"] - principals { - type = "AWS" - identifiers = ["arn:aws:iam::${var.aws_account_id}:root"] - } - condition { - test = "ArnLike" - variable = "aws:PrincipalArn" - values = ["arn:aws:iam::${var.aws_account_id}:role/${var.resource_prefix}-storage-credential"] - } - condition { - test = "StringEquals" - variable = "sts:ExternalId" - values = [var.databricks_account_id] - } - } +data "databricks_aws_unity_catalog_assume_role_policy" "external_location_example" { + aws_account_id = var.aws_account_id + role_name = "${var.resource_prefix}-storage-credential-example" + external_id = var.databricks_account_id } // Storage Credential Role resource "aws_iam_role" "storage_credential_role" { - name = "${var.resource_prefix}-storage-credential" - assume_role_policy = data.aws_iam_policy_document.passrole_for_storage_credential.json + name = "${var.resource_prefix}-storage-credential-example" + assume_role_policy = data.databricks_aws_unity_catalog_assume_role_policy.external_location_example.json tags = { - Name = "${var.resource_prefix}-storage_credential_role" + Name = "${var.resource_prefix}-storage-credential-example" + Project = var.resource_prefix } } - // Storage Credential Policy resource "aws_iam_role_policy" "storage_credential_policy" { - name = "${var.resource_prefix}-storage-credential-policy" + name = "${var.resource_prefix}-storage-credential-policy-example" role = aws_iam_role.storage_credential_role.id policy = jsonencode({ Version : "2012-10-17", Statement : [ @@ -76,7 +47,7 @@ resource "aws_iam_role_policy" "storage_credential_policy" { "sts:AssumeRole" ], "Resource" : [ - "arn:aws:iam::${var.aws_account_id}:role/${var.resource_prefix}-storage-credential" + "arn:aws:iam::${var.aws_account_id}:role/${var.resource_prefix}-storage-credential-example" ], "Effect" : "Allow" } @@ -91,7 +62,8 @@ resource "databricks_storage_credential" "external" { aws_iam_role { role_arn = aws_iam_role.storage_credential_role.arn } - depends_on = [aws_iam_role.storage_credential_role, time_sleep.wait_30_seconds] + isolation_mode = "ISOLATION_MODE_ISOLATED" + depends_on = [aws_iam_role.storage_credential_role, time_sleep.wait_30_seconds] } // External Location @@ -99,9 +71,9 @@ resource "databricks_external_location" "data_example" { name = "external-location-example" url = "s3://${var.read_only_data_bucket}/" credential_name = databricks_storage_credential.external.id - skip_validation = true read_only = true - comment = "Managed by TF" + comment = "Read only external location for ${var.read_only_data_bucket}" + isolation_mode = "ISOLATION_MODE_ISOLATED" } // External Location Grant diff --git a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf index 3a838a6..47bdf84 100644 --- a/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf +++ b/aws/tf/modules/sra/databricks_workspace/workspace_security_modules/uc_external_location/variables.tf @@ -1,12 +1,8 @@ -variable "databricks_account_id" { - type = string -} - variable "aws_account_id" { type = string } -variable "resource_prefix" { +variable "databricks_account_id" { type = string } @@ -16,4 +12,8 @@ variable "read_only_data_bucket" { variable "read_only_external_location_admin" { type = string +} + +variable "resource_prefix" { + type = string } \ No newline at end of file diff --git a/aws/tf/modules/sra/network.tf b/aws/tf/modules/sra/network.tf index cf13eb3..06d9448 100644 --- a/aws/tf/modules/sra/network.tf +++ b/aws/tf/modules/sra/network.tf @@ -25,6 +25,10 @@ module "vpc" { intra_subnet_names = [for az in var.availability_zones : format("%s-privatelink-%s", var.resource_prefix, az)] intra_subnets = var.privatelink_subnets_cidr + + tags = { + Project = var.resource_prefix + } } @@ -36,7 +40,7 @@ resource "aws_security_group" "sg" { depends_on = [module.vpc] dynamic "ingress" { - for_each = var.sg_ingress_protocol + for_each = ["tcp", "udp"] content { description = "Databricks - Workspace SG - Internode Communication" from_port = 0 @@ -47,7 +51,7 @@ resource "aws_security_group" "sg" { } dynamic "egress" { - for_each = var.sg_egress_protocol + for_each = ["tcp", "udp"] content { description = "Databricks - Workspace SG - Internode Communication" from_port = 0 @@ -80,6 +84,7 @@ resource "aws_security_group" "sg" { } } tags = { - Name = "${var.resource_prefix}-workspace-sg" + Name = "${var.resource_prefix}-workspace-sg" + Project = var.resource_prefix } } \ No newline at end of file diff --git a/aws/tf/modules/sra/privatelink.tf b/aws/tf/modules/sra/privatelink.tf index 5fc3ef4..a16d738 100644 --- a/aws/tf/modules/sra/privatelink.tf +++ b/aws/tf/modules/sra/privatelink.tf @@ -41,7 +41,8 @@ resource "aws_security_group" "privatelink" { } tags = { - Name = "${var.resource_prefix}-private-link-sg" + Name = "${var.resource_prefix}-private-link-sg", + Project = var.resource_prefix } } @@ -217,7 +218,7 @@ data "aws_iam_policy_document" "sts_vpc_endpoint_policy" { principals { type = "AWS" identifiers = [ - "arn:aws:iam::414351767826:user/databricks-datasets-readonly-user", + "arn:aws:iam::414351767826:user/databricks-datasets-readonly-user-prod", "414351767826" ] } @@ -261,7 +262,8 @@ module "vpc_endpoints" { route_table_ids = module.vpc[0].private_route_table_ids policy = var.enable_restrictive_s3_endpoint_boolean ? data.aws_iam_policy_document.s3_vpc_endpoint_policy[0].json : null tags = { - Name = "${var.resource_prefix}-s3-vpc-endpoint" + Name = "${var.resource_prefix}-s3-vpc-endpoint" + Project = var.resource_prefix } }, sts = { @@ -270,7 +272,8 @@ module "vpc_endpoints" { subnet_ids = module.vpc[0].intra_subnets policy = var.enable_restrictive_sts_endpoint_boolean ? data.aws_iam_policy_document.sts_vpc_endpoint_policy[0].json : null tags = { - Name = "${var.resource_prefix}-sts-vpc-endpoint" + Name = "${var.resource_prefix}-sts-vpc-endpoint" + Project = var.resource_prefix } }, kinesis-streams = { @@ -279,13 +282,11 @@ module "vpc_endpoints" { subnet_ids = module.vpc[0].intra_subnets policy = var.enable_restrictive_kinesis_endpoint_boolean ? data.aws_iam_policy_document.kinesis_vpc_endpoint_policy[0].json : null tags = { - Name = "${var.resource_prefix}-kinesis-vpc-endpoint" + Name = "${var.resource_prefix}-kinesis-vpc-endpoint" + Project = var.resource_prefix } } } - depends_on = [ - module.vpc, module.databricks_mws_workspace - ] } // Databricks REST endpoint - skipped in custom operation mode @@ -293,14 +294,15 @@ resource "aws_vpc_endpoint" "backend_rest" { count = var.operation_mode != "custom" ? 1 : 0 vpc_id = module.vpc[0].vpc_id - service_name = var.workspace_vpce_service + service_name = var.workspace[var.region] vpc_endpoint_type = "Interface" security_group_ids = [aws_security_group.privatelink[0].id] subnet_ids = module.vpc[0].intra_subnets private_dns_enabled = true depends_on = [module.vpc.vpc_id] tags = { - Name = "${var.resource_prefix}-databricks-backend-rest" + Name = "${var.resource_prefix}-databricks-backend-rest" + Project = var.resource_prefix } } @@ -309,13 +311,14 @@ resource "aws_vpc_endpoint" "backend_relay" { count = var.operation_mode != "custom" ? 1 : 0 vpc_id = module.vpc[0].vpc_id - service_name = var.relay_vpce_service + service_name = var.scc_relay[var.region] vpc_endpoint_type = "Interface" security_group_ids = [aws_security_group.privatelink[0].id] subnet_ids = module.vpc[0].intra_subnets private_dns_enabled = true depends_on = [module.vpc.vpc_id] tags = { - Name = "${var.resource_prefix}-databricks-backend-relay" + Name = "${var.resource_prefix}-databricks-backend-relay" + Project = var.resource_prefix } } \ No newline at end of file diff --git a/aws/tf/modules/sra/root_s3_bucket.tf b/aws/tf/modules/sra/root_s3_bucket.tf index 521910a..4c45c42 100644 --- a/aws/tf/modules/sra/root_s3_bucket.tf +++ b/aws/tf/modules/sra/root_s3_bucket.tf @@ -4,7 +4,8 @@ resource "aws_s3_bucket" "root_storage_bucket" { bucket = "${var.resource_prefix}-workspace-root-storage" force_destroy = true tags = { - Name = var.resource_prefix + Name = "${var.resource_prefix}-workspace-root-storage" + Project = var.resource_prefix } } @@ -17,7 +18,6 @@ resource "aws_s3_bucket_versioning" "root_bucket_versioning" { resource "aws_s3_bucket_server_side_encryption_configuration" "root_storage_bucket" { bucket = aws_s3_bucket.root_storage_bucket.bucket - rule { bucket_key_enabled = true apply_server_side_encryption_by_default { diff --git a/aws/tf/modules/sra/variables.tf b/aws/tf/modules/sra/variables.tf index 82cece4..86d5058 100644 --- a/aws/tf/modules/sra/variables.tf +++ b/aws/tf/modules/sra/variables.tf @@ -9,11 +9,6 @@ variable "aws_account_id" { sensitive = true } -variable "cmk_admin_arn" { - description = "Amazon Resource Name (ARN) of the CMK admin." - type = string -} - variable "client_id" { description = "Client ID for Databricks authentication." type = string @@ -26,6 +21,17 @@ variable "client_secret" { sensitive = true } +variable "cmk_admin_arn" { + description = "Amazon Resource Name (ARN) of the CMK admin." + type = string +} + +variable "compliance_security_profile_egress_ports" { + type = bool + description = "Add 2443 to security group configuration or nitro instance" + nullable = false +} + variable "custom_private_subnet_ids" { type = list(string) description = "List of custom private subnet IDs" @@ -36,7 +42,6 @@ variable "custom_relay_vpce_id" { description = "Custom Relay VPC Endpoint ID" } - variable "custom_sg_id" { type = string description = "Custom security group ID" @@ -52,16 +57,16 @@ variable "custom_workspace_vpce_id" { description = "Custom Workspace VPC Endpoint ID" } - variable "databricks_account_id" { description = "ID of the Databricks account." type = string sensitive = true } -variable "read_only_data_bucket" { - description = "S3 bucket for data storage." - type = string +variable "enable_admin_configs_boolean" { + type = bool + description = "Enable workspace configs" + nullable = false } variable "enable_audit_log_alerting" { @@ -78,13 +83,6 @@ variable "enable_cluster_boolean" { default = false } -variable "enable_read_only_external_location_boolean" { - description = "Flag to enable read only external location" - type = bool - sensitive = true - default = false -} - variable "enable_ip_boolean" { description = "Flag to enable IP-related configurations." type = bool @@ -99,8 +97,8 @@ variable "enable_logging_boolean" { default = false } -variable "enable_restrictive_root_bucket_boolean" { - description = "Flag to enable restrictive root bucket settings." +variable "enable_read_only_external_location_boolean" { + description = "Flag to enable read only external location" type = bool sensitive = true default = false @@ -112,6 +110,13 @@ variable "enable_restrictive_kinesis_endpoint_boolean" { default = false } +variable "enable_restrictive_root_bucket_boolean" { + description = "Flag to enable restrictive root bucket settings." + type = bool + sensitive = true + default = false +} + variable "enable_restrictive_s3_endpoint_boolean" { type = bool description = "Enable restrictive S3 endpoint boolean flag" @@ -124,7 +129,6 @@ variable "enable_restrictive_sts_endpoint_boolean" { default = false } - variable "enable_sat_boolean" { description = "Flag for a specific SAT (Service Access Token) configuration." type = bool @@ -144,18 +148,30 @@ variable "firewall_allow_list" { type = list(string) } -variable "firewall_protocol_deny_list" { - description = "Protocol list that the firewall should deny." - type = string -} - variable "firewall_subnets_cidr" { description = "CIDR blocks for firewall subnets." type = list(string) } -variable "hive_metastore_fqdn" { - type = string +variable "hms_fqdn" { + type = map(string) + default = { + "ap-northeast-1" = "mddx5a4bpbpm05.cfrfsun7mryq.ap-northeast-1.rds.amazonaws.com" + "ap-northeast-2" = "md1915a81ruxky5.cfomhrbro6gt.ap-northeast-2.rds.amazonaws.com" + "ap-south-1" = "mdjanpojt83v6j.c5jml0fhgver.ap-south-1.rds.amazonaws.com" + "ap-southeast-1" = "md1n4trqmokgnhr.csnrqwqko4ho.ap-southeast-1.rds.amazonaws.com" + "ap-southeast-2" = "mdnrak3rme5y1c.c5f38tyb1fdu.ap-southeast-2.rds.amazonaws.com" + "ca-central-1" = "md1w81rjeh9i4n5.co1tih5pqdrl.ca-central-1.rds.amazonaws.com" + "eu-central-1" = "mdv2llxgl8lou0.ceptxxgorjrc.eu-central-1.rds.amazonaws.com" + "eu-west-1" = "md15cf9e1wmjgny.cxg30ia2wqgj.eu-west-1.rds.amazonaws.com" + "eu-west-2" = "mdio2468d9025m.c6fvhwk6cqca.eu-west-2.rds.amazonaws.com" + "eu-west-3" = "metastorerds-dbconsolidationmetastore-asda4em2u6eg.c2ybp3dss6ua.eu-west-3.rds.amazonaws.com" + "sa-east-1" = "metastorerds-dbconsolidationmetastore-fqekf3pck8yw.cog1aduyg4im.sa-east-1.rds.amazonaws.com" + "us-east-1" = "mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com" + "us-east-2" = "md7wf1g369xf22.cluz8hwxjhb6.us-east-2.rds.amazonaws.com" + "us-west-1" = "mdzsbtnvk0rnce.c13weuwubexq.us-west-1.rds.amazonaws.com" + "us-west-2" = "mdpartyyphlhsp.caj77bnxuhme.us-west-2.rds.amazonaws.com" + } } variable "ip_addresses" { @@ -179,18 +195,6 @@ variable "operation_mode" { } } -variable "compliance_security_profile_egress_ports" { - type = bool - description = "Add 2443 to security group configuration or nitro instance" - nullable = false -} - -variable "enable_admin_configs_boolean" { - type = bool - description = "Enable workspace configs" - nullable = false -} - variable "private_subnets_cidr" { description = "CIDR blocks for private subnets." type = list(string) @@ -206,6 +210,16 @@ variable "public_subnets_cidr" { type = list(string) } +variable "read_only_data_bucket" { + description = "S3 bucket for data storage." + type = string +} + +variable "read_only_external_location_admin" { + description = "User to grant external location admin." + type = string +} + variable "region" { description = "AWS region code." type = string @@ -216,28 +230,33 @@ variable "region_name" { type = string } -variable "relay_vpce_service" { - description = "VPCE service for the secure cluster connectivity relay." - type = string -} - variable "resource_prefix" { description = "Prefix for the resource names." type = string } -variable "sg_egress_ports" { - description = "List of egress ports for security groups." - type = list(string) -} - -variable "sg_egress_protocol" { - description = "List of egress protocols for security groups." - type = list(string) +variable "scc_relay" { + type = map(string) + default = { + "ap-northeast-1" = "com.amazonaws.vpce.ap-northeast-1.vpce-svc-02aa633bda3edbec0" + "ap-northeast-2" = "com.amazonaws.vpce.ap-northeast-2.vpce-svc-0dc0e98a5800db5c4" + "ap-south-1" = "com.amazonaws.vpce.ap-south-1.vpce-svc-03fd4d9b61414f3de" + "ap-southeast-1" = "com.amazonaws.vpce.ap-southeast-1.vpce-svc-0557367c6fc1a0c5c" + "ap-southeast-2" = "com.amazonaws.vpce.ap-southeast-2.vpce-svc-0b4a72e8f825495f6" + "ca-central-1" = "com.amazonaws.vpce.ca-central-1.vpce-svc-0c4e25bdbcbfbb684" + "eu-central-1" = "com.amazonaws.vpce.eu-central-1.vpce-svc-08e5dfca9572c85c4" + "eu-west-1" = "com.amazonaws.vpce.eu-west-1.vpce-svc-09b4eb2bc775f4e8c" + "eu-west-2" = "com.amazonaws.vpce.eu-west-2.vpce-svc-05279412bf5353a45" + "eu-west-3" = "com.amazonaws.vpce.eu-west-3.vpce-svc-005b039dd0b5f857d" + "sa-east-1" = "com.amazonaws.vpce.sa-east-1.vpce-svc-0e61564963be1b43f" + "us-east-1" = "com.amazonaws.vpce.us-east-1.vpce-svc-00018a8c3ff62ffdf" + "us-east-2" = "com.amazonaws.vpce.us-east-2.vpce-svc-090a8fab0d73e39a6" + "us-west-2" = "com.amazonaws.vpce.us-west-2.vpce-svc-0158114c0c730c3bb" + } } -variable "sg_ingress_protocol" { - description = "List of ingress protocols for security groups." +variable "sg_egress_ports" { + description = "List of egress ports for security groups." type = list(string) } @@ -247,8 +266,8 @@ variable "user_workspace_admin" { nullable = false } -variable "read_only_external_location_admin" { - description = "User to grant external location admin." +variable "user_workspace_catalog_admin" { + description = "Admin for the workspace catalog" type = string } @@ -257,17 +276,23 @@ variable "vpc_cidr_range" { type = string } -variable "workspace_catalog_admin" { - description = "Admin for the workspace catalog" - type = string -} - -variable "workspace_vpce_service" { - description = "VPCE service for the workspace REST API endpoint." - type = string +variable "workspace" { + type = map(string) + default = { + "ap-northeast-1" = "com.amazonaws.vpce.ap-northeast-1.vpce-svc-02691fd610d24fd64" + "ap-northeast-2" = "com.amazonaws.vpce.ap-northeast-2.vpce-svc-0babb9bde64f34d7e" + "ap-south-1" = "com.amazonaws.vpce.ap-south-1.vpce-svc-0dbfe5d9ee18d6411" + "ap-southeast-1" = "com.amazonaws.vpce.ap-southeast-1.vpce-svc-02535b257fc253ff4" + "ap-southeast-2" = "com.amazonaws.vpce.ap-southeast-2.vpce-svc-0b87155ddd6954974" + "ca-central-1" = "com.amazonaws.vpce.ca-central-1.vpce-svc-0205f197ec0e28d65" + "eu-central-1" = "com.amazonaws.vpce.eu-central-1.vpce-svc-081f78503812597f7" + "eu-west-1" = "com.amazonaws.vpce.eu-west-1.vpce-svc-0da6ebf1461278016" + "eu-west-2" = "com.amazonaws.vpce.eu-west-2.vpce-svc-01148c7cdc1d1326c" + "eu-west-3" = "com.amazonaws.vpce.eu-west-3.vpce-svc-008b9368d1d011f37" + "sa-east-1" = "com.amazonaws.vpce.sa-east-1.vpce-svc-0bafcea8cdfe11b66" + "us-east-1" = "com.amazonaws.vpce.us-east-1.vpce-svc-09143d1e626de2f04" + "us-east-2" = "com.amazonaws.vpce.us-east-2.vpce-svc-041dc2b4d7796b8d3" + "us-west-2" = "com.amazonaws.vpce.us-west-2.vpce-svc-0129f463fcfbc46c5" + #"us-west-1" = "" + } } - -variable "workspace_admin_service_principal_name" { - description = "Service principle name" - type = string -} \ No newline at end of file diff --git a/aws/tf/provider.tf b/aws/tf/provider.tf index 0c7c62d..dbfc9a3 100644 --- a/aws/tf/provider.tf +++ b/aws/tf/provider.tf @@ -2,7 +2,7 @@ terraform { required_providers { databricks = { source = "databricks/databricks" - version = "~> 1.46.0" + version = " 1.50.0" } aws = { source = "hashicorp/aws" diff --git a/aws/tf/sra.tf b/aws/tf/sra.tf index c07cf69..e094ffa 100644 --- a/aws/tf/sra.tf +++ b/aws/tf/sra.tf @@ -5,7 +5,7 @@ module "SRA" { aws = aws } - // Common Authentication Variables + // REQUIRED - Authentication: databricks_account_id = var.databricks_account_id client_id = var.client_id client_secret = var.client_secret @@ -13,47 +13,40 @@ module "SRA" { region = var.region region_name = var.region_name[var.region] - // Naming and Tagging Variables: + // REQUIRED - Naming and Tagging: resource_prefix = var.resource_prefix - // Required Variables: - workspace_catalog_admin = null // Workspace catalog admin email. - user_workspace_admin = null // Workspace admin user email. - operation_mode = "sandbox" // Operation mode (sandbox, custom, firewall, isolated). - workspace_admin_service_principal_name = "sra-example-sp" // Creates an example admin SP for automation use cases. - metastore_exists = false // If a regional metastore exists set to true. If there are multiple regional metastores, you can comment out "uc_init" and add the metastore ID directly in to the module call for "uc_assignment". + // REQUIRED - Workspace and Unity Catalog: + user_workspace_admin = null // Workspace admin user email. + user_workspace_catalog_admin = null // Workspace catalog admin email. + operation_mode = "isolated" // Operation mode (sandbox, custom, firewall, isolated), see README.md for more information. + metastore_exists = false // If a regional metastore exists set to true. If there are multiple regional metastores, you can comment out "uc_init" and add the metastore ID directly in to the module call for "uc_assignment". - // AWS Specific Variables: - cmk_admin_arn = null // CMK admin ARN, defaults to the AWS account root user. + // REQUIRED - AWS Infrastructure: + cmk_admin_arn = null // CMK admin ARN, defaults to the AWS account root user. vpc_cidr_range = "10.0.0.0/18" // Please re-define the subsequent subnet ranges if the VPC CIDR range is updated. private_subnets_cidr = ["10.0.0.0/22", "10.0.4.0/22", "10.0.8.0/22"] privatelink_subnets_cidr = ["10.0.28.0/26", "10.0.28.64/26", "10.0.28.128/26"] availability_zones = [data.aws_availability_zones.available.names[0], data.aws_availability_zones.available.names[1], data.aws_availability_zones.available.names[2]] sg_egress_ports = [443, 3306, 6666, 8443, 8444, 8445, 8446, 8447, 8448, 8449, 8450, 8451] compliance_security_profile_egress_ports = false // Set to true to enable compliance security profile related egress ports (2443) - sg_ingress_protocol = ["tcp", "udp"] - sg_egress_protocol = ["tcp", "udp"] - relay_vpce_service = var.scc_relay[var.region] - workspace_vpce_service = var.workspace[var.region] - // Operation Mode Specific Variables: - // Sandbox and Firewall Modes + // Operation Mode Specific: + // Sandbox and Firewall Operation Mode: public_subnets_cidr = ["10.0.29.0/26", "10.0.29.64/26", "10.0.29.128/26"] - // Firewall Mode Specific: - firewall_subnets_cidr = ["10.0.33.0/26", "10.0.33.64/26", "10.0.33.128/26"] - firewall_allow_list = [".pypi.org", ".cran.r-project.org", ".pythonhosted.org", ".spark-packages.org", ".maven.org", "maven.apache.org", ".storage-download.googleapis.com"] - firewall_protocol_deny_list = "IP" - hive_metastore_fqdn = "mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com" // https://docs.databricks.com/en/resources/supported-regions.html#rds-addresses-for-legacy-hive-metastore + // Firewall Operation Mode: + firewall_subnets_cidr = ["10.0.33.0/26", "10.0.33.64/26", "10.0.33.128/26"] + firewall_allow_list = [".pypi.org", ".cran.r-project.org", ".pythonhosted.org", ".spark-packages.org", ".maven.org", "maven.apache.org", ".storage-download.googleapis.com"] - // Custom Mode Specific: + // Custom Operation Mode: custom_vpc_id = null custom_private_subnet_ids = null // List of custom private subnet IDs required. custom_sg_id = null custom_relay_vpce_id = null custom_workspace_vpce_id = null - // Optional Features: + // OPTIONAL - Examples, Workspace Hardening, and Solution Accelerators: enable_read_only_external_location_boolean = false // Set to true to enable a read-only external location. read_only_data_bucket = null // S3 bucket name for read-only data. read_only_external_location_admin = null // Admin for the external location. @@ -62,15 +55,15 @@ module "SRA" { enable_admin_configs_boolean = false // Set to true to enable optional admin configurations. enable_logging_boolean = false // Set to true to enable log delivery and creation of related assets (e.g. S3 bucket and IAM role) - enable_restrictive_root_bucket_boolean = false - enable_restrictive_s3_endpoint_boolean = false - enable_restrictive_sts_endpoint_boolean = false - enable_restrictive_kinesis_endpoint_boolean = false + enable_restrictive_root_bucket_boolean = false // Set to true to enable a restrictive root bucket policy, this is subject to change and may cause unexpected issues in the event of a change. + enable_restrictive_s3_endpoint_boolean = false // Set to true to enable a restrictive S3 endpoint policy, this is subject to change and may cause unexpected issues in the event of a change. + enable_restrictive_sts_endpoint_boolean = false // Set to true to enable a restrictive STS endpoint policy, this is subject to change and may cause unexpected issues in the event of a change. + enable_restrictive_kinesis_endpoint_boolean = false // Set to true to enable a restrictive Kinesis endpoint policy, this is subject to change and may cause unexpected issues in the event of a change. enable_ip_boolean = false // Set to true to enable IP access list. ip_addresses = ["X.X.X.X", "X.X.X.X/XX", "X.X.X.X/XX"] // Specify IP addresses for access. - enable_system_tables_schema_boolean = false // Set to true to enable system table schemas (Public Preview). + enable_system_tables_schema_boolean = false // Set to true to enable system table schemas enable_sat_boolean = false // Set to true to enable Security Analysis Tool. https://github.com/databricks-industry-solutions/security-analysis-tool enable_audit_log_alerting = false // Set to true to create 40+ queries for audit log alerting based on user activity. https://github.com/andyweaves/system-tables-audit-logs diff --git a/aws/tf/variables.tf b/aws/tf/variables.tf index 62897ec..4b0e753 100644 --- a/aws/tf/variables.tf +++ b/aws/tf/variables.tf @@ -1,3 +1,7 @@ +data "aws_availability_zones" "available" { + state = "available" +} + variable "aws_account_id" { description = "ID of the AWS account." type = string @@ -55,50 +59,4 @@ variable "region_name" { variable "resource_prefix" { description = "Prefix for the resource names." type = string -} - -data "aws_availability_zones" "available" { - state = "available" -} - -variable "workspace" { - type = map(string) - default = { - "ap-northeast-1" = "com.amazonaws.vpce.ap-northeast-1.vpce-svc-02691fd610d24fd64" - "ap-northeast-2" = "com.amazonaws.vpce.ap-northeast-2.vpce-svc-0babb9bde64f34d7e" - "ap-south-1" = "com.amazonaws.vpce.ap-south-1.vpce-svc-0dbfe5d9ee18d6411" - "ap-southeast-1" = "com.amazonaws.vpce.ap-southeast-1.vpce-svc-02535b257fc253ff4" - "ap-southeast-2" = "com.amazonaws.vpce.ap-southeast-2.vpce-svc-0b87155ddd6954974" - "ca-central-1" = "com.amazonaws.vpce.ca-central-1.vpce-svc-0205f197ec0e28d65" - "eu-central-1" = "com.amazonaws.vpce.eu-central-1.vpce-svc-081f78503812597f7" - "eu-west-1" = "com.amazonaws.vpce.eu-west-1.vpce-svc-0da6ebf1461278016" - "eu-west-2" = "com.amazonaws.vpce.eu-west-2.vpce-svc-01148c7cdc1d1326c" - "eu-west-3" = "com.amazonaws.vpce.eu-west-3.vpce-svc-008b9368d1d011f37" - "sa-east-1" = "com.amazonaws.vpce.sa-east-1.vpce-svc-0bafcea8cdfe11b66" - "us-east-1" = "com.amazonaws.vpce.us-east-1.vpce-svc-09143d1e626de2f04" - "us-east-2" = "com.amazonaws.vpce.us-east-2.vpce-svc-041dc2b4d7796b8d3" - "us-west-2" = "com.amazonaws.vpce.us-west-2.vpce-svc-0129f463fcfbc46c5" - #"us-west-1" = "" - } -} - -variable "scc_relay" { - type = map(string) - default = { - "ap-northeast-1" = "com.amazonaws.vpce.ap-northeast-1.vpce-svc-02aa633bda3edbec0" - "ap-northeast-2" = "com.amazonaws.vpce.ap-northeast-2.vpce-svc-0dc0e98a5800db5c4" - "ap-south-1" = "com.amazonaws.vpce.ap-south-1.vpce-svc-03fd4d9b61414f3de" - "ap-southeast-1" = "com.amazonaws.vpce.ap-southeast-1.vpce-svc-0557367c6fc1a0c5c" - "ap-southeast-2" = "com.amazonaws.vpce.ap-southeast-2.vpce-svc-0b4a72e8f825495f6" - "ca-central-1" = "com.amazonaws.vpce.ca-central-1.vpce-svc-0c4e25bdbcbfbb684" - "eu-central-1" = "com.amazonaws.vpce.eu-central-1.vpce-svc-08e5dfca9572c85c4" - "eu-west-1" = "com.amazonaws.vpce.eu-west-1.vpce-svc-09b4eb2bc775f4e8c" - "eu-west-2" = "com.amazonaws.vpce.eu-west-2.vpce-svc-05279412bf5353a45" - "eu-west-3" = "com.amazonaws.vpce.eu-west-3.vpce-svc-005b039dd0b5f857d" - "sa-east-1" = "com.amazonaws.vpce.sa-east-1.vpce-svc-0e61564963be1b43f" - "us-east-1" = "com.amazonaws.vpce.us-east-1.vpce-svc-00018a8c3ff62ffdf" - "us-east-2" = "com.amazonaws.vpce.us-east-2.vpce-svc-090a8fab0d73e39a6" - "us-west-2" = "com.amazonaws.vpce.us-west-2.vpce-svc-0158114c0c730c3bb" - #"us-west-1" = "" - } } \ No newline at end of file