Merge branch 'main' into aastha/upgrade-google-module

CHANGELOG.md

@@ -2,6 +2,34 @@
 
 All notable changes to this project will be documented in this file.
 
+### [5.1.3](https://github.com/wandb/terraform-google-wandb/compare/v5.1.2...v5.1.3) (2024-07-25)
+
+
+### Bug Fixes
+
+* Typo on SA Member ([#155](https://github.com/wandb/terraform-google-wandb/issues/155)) ([2262da2](https://github.com/wandb/terraform-google-wandb/commit/2262da2f1a36647194c9f8292814798f15cb33a1))
+
+### [5.1.2](https://github.com/wandb/terraform-google-wandb/compare/v5.1.1...v5.1.2) (2024-07-23)
+
+
+### Bug Fixes
+
+* Correct Encryption Logic ([#154](https://github.com/wandb/terraform-google-wandb/issues/154)) ([e68805c](https://github.com/wandb/terraform-google-wandb/commit/e68805c0eb7115f3ff13a42d80fdefa0d966024c))
+
+### [5.1.1](https://github.com/wandb/terraform-google-wandb/compare/v5.1.0...v5.1.1) (2024-07-23)
+
+
+### Bug Fixes
+
+* Tier typo/mistake ([#153](https://github.com/wandb/terraform-google-wandb/issues/153)) ([5d632e4](https://github.com/wandb/terraform-google-wandb/commit/5d632e4408d91674f1ff33ebae49e6b583e91d72))
+
+## [5.1.0](https://github.com/wandb/terraform-google-wandb/compare/v5.0.1...v5.1.0) (2024-07-23)
+
+
+### Features
+
+* Added support for encrypting the database and bucket with CMK ([#100](https://github.com/wandb/terraform-google-wandb/issues/100)) ([7802e3c](https://github.com/wandb/terraform-google-wandb/commit/7802e3ce1f227f3e641d2e1bdb6c01db4de5cac9))
+
 ### [5.0.1](https://github.com/wandb/terraform-google-wandb/compare/v5.0.0...v5.0.1) (2024-07-22)
 
 

README.md

@@ -81,6 +81,8 @@ resources that lack official modules.
 | <a name="module_database"></a> [database](#module\_database) | ./modules/database | n/a |
 | <a name="module_gke_app"></a> [gke\_app](#module\_gke\_app) | wandb/wandb/kubernetes | 1.14.1 |
 | <a name="module_kms"></a> [kms](#module\_kms) | ./modules/kms | n/a |
+| <a name="module_kms_default_bucket"></a> [kms\_default\_bucket](#module\_kms\_default\_bucket) | ./modules/kms | n/a |
+| <a name="module_kms_default_sql"></a> [kms\_default\_sql](#module\_kms\_default\_sql) | ./modules/kms | n/a |
 | <a name="module_networking"></a> [networking](#module\_networking) | ./modules/networking | n/a |
 | <a name="module_private_link"></a> [private\_link](#module\_private\_link) | ./modules/private_link | n/a |
 | <a name="module_project_factory_project_services"></a> [project\_factory\_project\_services](#module\_project\_factory\_project\_services) | terraform-google-modules/project-factory/google//modules/project_services | ~> 15.0 |
@@ -104,13 +106,17 @@ resources that lack official modules.
 | <a name="input_allowed_inbound_cidrs"></a> [allowed\_inbound\_cidrs](#input\_allowed\_inbound\_cidrs) | Which IPv4 addresses/ranges to allow access. This must be explicitly provided, and by default is set to ["*"] | `list(string)` | <pre>[<br>  "*"<br>]</pre> | no |
 | <a name="input_allowed_project_names"></a> [allowed\_project\_names](#input\_allowed\_project\_names) | A map of allowed projects where each key is a project number and the value is the connection limit. | `map(number)` | `{}` | no |
 | <a name="input_app_wandb_env"></a> [app\_wandb\_env](#input\_app\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
+| <a name="input_bucket_default_encryption"></a> [bucket\_default\_encryption](#input\_bucket\_default\_encryption) | Boolean to determine if a default bucket encryption key should be used. If true, a default key will be created. Takes precedence over `bucket_kms_key_id`. | `bool` | `false` | no |
+| <a name="input_bucket_kms_key_id"></a> [bucket\_kms\_key\_id](#input\_bucket\_kms\_key\_id) | ID of the customer-provided bucket KMS key. | `string` | `null` | no |
+| <a name="input_bucket_location"></a> [bucket\_location](#input\_bucket\_location) | Location of the bucket (US, EU, ASIA) | `string` | `"US"` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Use an existing bucket. | `string` | `""` | no |
 | <a name="input_create_private_link"></a> [create\_private\_link](#input\_create\_private\_link) | Whether to create a private link service. | `bool` | `false` | no |
 | <a name="input_create_redis"></a> [create\_redis](#input\_create\_redis) | Boolean indicating whether to provision an redis instance (true) or not (false). | `bool` | `false` | no |
 | <a name="input_create_workload_identity"></a> [create\_workload\_identity](#input\_create\_workload\_identity) | Flag to indicate whether to create a workload identity for the service account. | `bool` | `false` | no |
 | <a name="input_database_machine_type"></a> [database\_machine\_type](#input\_database\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"db-n1-standard-2"` | no |
 | <a name="input_database_sort_buffer_size"></a> [database\_sort\_buffer\_size](#input\_database\_sort\_buffer\_size) | Specifies the sort\_buffer\_size value to set for the database | `number` | `67108864` | no |
 | <a name="input_database_version"></a> [database\_version](#input\_database\_version) | Version for MySQL | `string` | `"MYSQL_8_0_31"` | no |
+| <a name="input_db_kms_key_id"></a> [db\_kms\_key\_id](#input\_db\_kms\_key\_id) | ID of the customer-provided SQL KMS key. | `string` | `null` | no |
 | <a name="input_deletion_protection"></a> [deletion\_protection](#input\_deletion\_protection) | If the instance should have deletion protection enabled. The database / Bucket can't be deleted when this value is set to `true`. | `bool` | `true` | no |
 | <a name="input_disable_code_saving"></a> [disable\_code\_saving](#input\_disable\_code\_saving) | Boolean indicating if code saving is disabled | `bool` | `false` | no |
 | <a name="input_domain_name"></a> [domain\_name](#input\_domain\_name) | Domain for accessing the Weights & Biases UI. | `string` | `null` | no |
@@ -137,6 +143,7 @@ resources that lack official modules.
 | <a name="input_resource_limits"></a> [resource\_limits](#input\_resource\_limits) | Specifies the resource limits for the wandb deployment | `map(string)` | <pre>{<br>  "cpu": null,<br>  "memory": null<br>}</pre> | no |
 | <a name="input_resource_requests"></a> [resource\_requests](#input\_resource\_requests) | Specifies the resource requests for the wandb deployment | `map(string)` | <pre>{<br>  "cpu": "2000m",<br>  "memory": "2G"<br>}</pre> | no |
 | <a name="input_size"></a> [size](#input\_size) | Deployment size for the instance | `string` | `null` | no |
+| <a name="input_sql_default_encryption"></a> [sql\_default\_encryption](#input\_sql\_default\_encryption) | Boolean to determine if a default SQL encryption key should be used. If true, a default key will be created. Takes precedence over `db_kms_key_id`. | `bool` | `false` | no |
 | <a name="input_ssl"></a> [ssl](#input\_ssl) | Enable SSL certificate | `bool` | `true` | no |
 | <a name="input_stackdriver_sa_name"></a> [stackdriver\_sa\_name](#input\_stackdriver\_sa\_name) | n/a | `string` | `"wandb-stackdriver"` | no |
 | <a name="input_subdomain"></a> [subdomain](#input\_subdomain) | Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route. | `string` | `null` | no |

main.tf

@@ -54,8 +54,26 @@ module "kms" {
   deletion_protection = var.deletion_protection
 }
 
+module "kms_default_bucket" {
+  count                          = var.bucket_default_encryption ? 1 : 0
+  source                         = "./modules/kms"
+  namespace                      = var.namespace
+  deletion_protection            = var.deletion_protection
+  key_location                   = lower(var.bucket_location)
+  bind_pubsub_service_to_kms_key = false
+}
+
+module "kms_default_sql" {
+  count                          = var.sql_default_encryption ? 1 : 0
+  source                         = "./modules/kms"
+  namespace                      = var.namespace
+  deletion_protection            = var.deletion_protection
+  key_location                   = data.google_client_config.current.region
+  bind_pubsub_service_to_kms_key = false
+}
 locals {
-  crypto_key = var.use_internal_queue ? null : module.kms[0].crypto_key
+  default_bucket_key = length(module.kms_default_bucket) > 0 ? module.kms_default_bucket[0].crypto_key.id : var.bucket_kms_key_id
+  default_sql_key    = length(module.kms_default_sql) > 0 ? module.kms_default_sql[0].crypto_key.id : var.db_kms_key_id
 }
 
 module "storage" {
@@ -64,13 +82,14 @@ module "storage" {
   namespace = var.namespace
   labels    = var.labels
 
-  create_queue    = !var.use_internal_queue
-  bucket_location = "US"
-  service_account = module.service_accounts.service_account
-  crypto_key      = local.crypto_key
+  create_queue      = !var.use_internal_queue
+  bucket_location   = var.bucket_location
+  service_account   = module.service_accounts.service_account
+  bucket_crypto_key = local.default_bucket_key
+  crypto_key        = var.use_internal_queue ? null : module.kms[0].crypto_key
 
   deletion_protection = var.deletion_protection
-  depends_on          = [module.project_factory_project_services]
+  depends_on          = [module.project_factory_project_services, module.kms_default_bucket]
 }
 
 module "networking" {
@@ -109,8 +128,7 @@ module "app_lb" {
   service_account       = module.service_accounts.service_account
   labels                = var.labels
   allowed_inbound_cidrs = var.allowed_inbound_cidrs
-
-  depends_on = [module.project_factory_project_services, module.app_gke]
+  depends_on            = [module.project_factory_project_services, module.app_gke]
 }
 
 module "database" {
@@ -123,7 +141,8 @@ module "database" {
   network_connection  = local.network_connection
   deletion_protection = var.deletion_protection
   labels              = var.labels
-  depends_on          = [module.project_factory_project_services]
+  crypto_key          = local.default_sql_key
+  depends_on          = [module.project_factory_project_services, module.kms_default_sql]
 }
 
 module "redis" {
@@ -134,8 +153,10 @@ module "redis" {
   memory_size_gb    = coalesce(try(local.deployment_size[var.size].cache, 6))
   network           = local.network
   reserved_ip_range = var.redis_reserved_ip_range
-  labels            = var.labels
   tier              = var.redis_tier
+  labels            = var.labels
+  crypto_key        = local.default_sql_key
+  depends_on        = [module.project_factory_project_services, module.kms_default_sql]
 }
 
 locals {

modules/database/main.tf

@@ -25,11 +25,16 @@ locals {
   master_instance_name = "${var.namespace}-${random_pet.mysql.id}"
 }
 
+data "google_project" "default" {
+}
+
 resource "google_sql_database_instance" "default" {
   name                = local.master_instance_name
   database_version    = var.database_version
   deletion_protection = var.deletion_protection
 
+  encryption_key_name = var.crypto_key
+
   settings {
     tier                        = var.tier
     availability_type           = var.availability_type

modules/database/variables.tf

@@ -30,6 +30,7 @@ variable "availability_type" {
   default = "REGIONAL"
 }
 
+
 variable "maintenance_window_day" {
   description = "The day of week (1-7) for the master instance maintenance."
   type        = number
@@ -72,3 +73,9 @@ variable "force_ssl" {
   type        = bool
   default     = false
 }
+
+variable "crypto_key" {
+  type        = string
+  default     = null
+  description = "Key used to encrypt and decrypt database."
+}

modules/kms/main.tf

@@ -5,7 +5,7 @@ resource "random_pet" "key_ring" {
 
 resource "google_kms_key_ring" "default" {
   name     = "${var.namespace}-${random_pet.key_ring.id}"
-  location = "global"
+  location =  var.key_location
 }
 
 
@@ -28,21 +28,27 @@ resource "google_kms_crypto_key" "default" {
 
 data "google_project" "project" {}
 
+resource "google_project_service_identity" "gcp_sa_cloud_sql" {
+  provider = google-beta
+  project  = data.google_project.project.project_id
+  service  = "sqladmin.googleapis.com"
+}
+
 resource "google_project_service_identity" "pubsub" {
+  count    = var.bind_pubsub_service_to_kms_key ? 1 : 0
   provider = google-beta
   project  = data.google_project.project.project_id
   service  = "pubsub.googleapis.com"
 }
 
-# PubSub service account must have roles/cloudkms.cryptoKeyEncrypterDecrypter to
-# use pubsub topic encryption.
-# https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic#kms_key_name
 resource "google_kms_crypto_key_iam_member" "pubsub_service_access" {
+  count         = var.bind_pubsub_service_to_kms_key ? 1 : 0
   crypto_key_id = google_kms_crypto_key.default.id
   role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
-  member        = "serviceAccount:${google_project_service_identity.pubsub.email}"
+  member        = "serviceAccount:${google_project_service_identity.pubsub[0].email}"
 }
 
+
 # Enable notifications by giving the correct IAM permission to the unique
 # service account.
 data "google_storage_project_service_account" "default" {
@@ -52,4 +58,18 @@ resource "google_kms_crypto_key_iam_member" "storage_service_access" {
   crypto_key_id = google_kms_crypto_key.default.id
   role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
   member        = "serviceAccount:${data.google_storage_project_service_account.default.email_address}"
+}
+
+data "google_storage_project_service_account" "gcs_account" {
+}
+
+resource "google_kms_crypto_key_iam_binding" "crypto_key" {
+
+  crypto_key_id = google_kms_crypto_key.default.id
+  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
+  members = [
+    "serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}",
+    "serviceAccount:${google_project_service_identity.gcp_sa_cloud_sql.email}",
+    "serviceAccount:service-${data.google_project.project.number}@cloud-redis.iam.gserviceaccount.com"
+  ]
 }

modules/kms/outputs.tf

@@ -1,3 +1,6 @@
 output "crypto_key" {
   value = google_kms_crypto_key.default
+}
+output "google_kms_crypto_key_iam_binding" {
+ value = google_kms_crypto_key_iam_binding.crypto_key
 }

modules/kms/variables.tf

@@ -7,4 +7,16 @@ variable "deletion_protection" {
   description = "If the instance should have deletion protection enabled. The database / Bucket can't be deleted when this value is set to `true`."
   type        = bool
   default     = true
+}
+
+variable "key_location" {
+  type        = string
+  description = "Location where the KMS key will be created."
+  default = "global"
+}
+
+variable "bind_pubsub_service_to_kms_key" {
+  type        = bool
+  description = "Whether to bind the Pub/Sub service account to the KMS key for encrypter/decrypter access."
+  default = true
 }

modules/redis/main.tf

@@ -6,6 +6,7 @@ resource "google_redis_instance" "default" {
   display_name   = "${var.namespace} W&B Instance"
   tier           = var.tier
   memory_size_gb = var.memory_size_gb
+  customer_managed_key = var.crypto_key
 
   location_id             = data.google_compute_zones.available.names.0
   alternative_location_id = data.google_compute_zones.available.names.1

modules/redis/variables.tf

@@ -27,4 +27,10 @@ variable "reserved_ip_range" {
 variable "tier" {
   type        = string
   description = "Specifies the tier for this Redis instance"
+}
+
+variable "crypto_key" {
+  type        = string
+  default     = null
+  description = "Key used to encrypt and decrypt redis."
 }

modules/service_accounts/main.tf

@@ -101,7 +101,7 @@ resource "google_project_iam_member" "storage" {
 resource "google_storage_bucket_iam_member" "gcs_admin" {
   count  = var.create_workload_identity == true && var.bucket_name != "" ? 1 : 0
   bucket = var.bucket_name
-  member = google_service_account.kms_gcs_sa[0].email
+  member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}"
   role   = "roles/storage.objectAdmin"
 }
 

modules/storage/bucket/main.tf

@@ -5,6 +5,8 @@ locals {
 resource "random_pet" "file_storage" {
   length = 2
 }
+data "google_project" "default" {
+}
 
 resource "google_storage_bucket" "file_storage" {
   name     = "${var.namespace}-${random_pet.file_storage.id}"
@@ -15,6 +17,13 @@ resource "google_storage_bucket" "file_storage" {
   force_destroy               = !var.deletion_protection
 
   labels = var.labels
+
+   dynamic "encryption" {
+    for_each = var.crypto_key != null ? [1] : []
+    content {
+      default_kms_key_name = var.crypto_key
+    }
+  }
 
   cors {
     origin          = ["*"]
@@ -28,4 +37,4 @@ resource "google_storage_bucket_iam_member" "object_admin" {
   bucket = google_storage_bucket.file_storage.name
   member = local.sa_member
   role   = "roles/storage.objectAdmin"
-}
+}

modules/storage/bucket/variables.tf

@@ -30,3 +30,9 @@ variable "project_id" {
   default     = null
   description = "The project ID to deploy to. If unset, the provider's default project is used."
 }
+
+variable "crypto_key" {
+  type        = string
+  default     = null
+  description = "Key used to encrypt and decrypt pubsub."
+}

modules/storage/main.tf

@@ -1,25 +1,22 @@
 module "bucket" {
-  source     = "./bucket"
-  project_id = var.project_id
-  namespace  = var.namespace
-  labels     = var.labels
-
-  bucket_location = var.bucket_location
-  service_account = var.service_account
-
+  source              = "./bucket"
+  project_id          = var.project_id
+  namespace           = var.namespace
+  labels              = var.labels
+  bucket_location     = var.bucket_location
+  service_account     = var.service_account
   deletion_protection = var.deletion_protection
+  crypto_key          = var.bucket_crypto_key
 }
 
 module "pubsub" {
   count = var.create_queue ? 1 : 0
 
-  source    = "./pubsub"
-  namespace = var.namespace
-  labels    = var.labels
-
-  bucket          = module.bucket.bucket_name
-  service_account = var.service_account
-  crypto_key      = var.crypto_key
-
+  source              = "./pubsub"
+  namespace           = var.namespace
+  labels              = var.labels
+  bucket              = module.bucket.bucket_name
+  service_account     = var.service_account
+  crypto_key          = var.crypto_key
   deletion_protection = var.deletion_protection
 }

modules/storage/variables.tf

@@ -36,6 +36,12 @@ variable "crypto_key" {
   description = "Key used to encrypt and decrypt pubsub."
 }
 
+variable "bucket_crypto_key" {
+  type        = string
+  default     = null
+  description = "Key used to encrypt and decrypt storage bucket."
+}
+
 variable "project_id" {
   type        = string
   default     = null