feat: Added support for GCP Private Service Connect

README.md

@@ -61,15 +61,16 @@ resources that lack official modules.
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | ~> 1.0 |
-| <a name="requirement_google"></a> [google](#requirement\_google) | ~> 4.82 |
+| <a name="requirement_google"></a> [google](#requirement\_google) | ~> 5.30 |
 | <a name="requirement_helm"></a> [helm](#requirement\_helm) | ~> 2.10 |
 | <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | ~> 2.23 |
+| <a name="requirement_time"></a> [time](#requirement\_time) | 0.11.2 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_google"></a> [google](#provider\_google) | ~> 4.82 |
+| <a name="provider_google"></a> [google](#provider\_google) | ~> 5.30 |
 
 ## Modules
 
@@ -81,9 +82,11 @@ resources that lack official modules.
 | <a name="module_gke_app"></a> [gke\_app](#module\_gke\_app) | wandb/wandb/kubernetes | 1.14.1 |
 | <a name="module_kms"></a> [kms](#module\_kms) | ./modules/kms | n/a |
 | <a name="module_networking"></a> [networking](#module\_networking) | ./modules/networking | n/a |
-| <a name="module_project_factory_project_services"></a> [project\_factory\_project\_services](#module\_project\_factory\_project\_services) | terraform-google-modules/project-factory/google//modules/project_services | ~> 13.0 |
+| <a name="module_private_link"></a> [private\_link](#module\_private\_link) | ./modules/private_link | n/a |
+| <a name="module_project_factory_project_services"></a> [project\_factory\_project\_services](#module\_project\_factory\_project\_services) | terraform-google-modules/project-factory/google//modules/project_services | ~> 14.0 |
 | <a name="module_redis"></a> [redis](#module\_redis) | ./modules/redis | n/a |
 | <a name="module_service_accounts"></a> [service\_accounts](#module\_service\_accounts) | ./modules/service_accounts | n/a |
+| <a name="module_sleep"></a> [sleep](#module\_sleep) | matti/resource/shell | 1.5.0 |
 | <a name="module_storage"></a> [storage](#module\_storage) | ./modules/storage | n/a |
 | <a name="module_wandb"></a> [wandb](#module\_wandb) | wandb/wandb/helm | 1.2.0 |
 
@@ -92,14 +95,17 @@ resources that lack official modules.
 | Name | Type |
 |------|------|
 | [google_client_config.current](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_config) | data source |
+| [google_compute_forwarding_rules.all](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_forwarding_rules) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_allowed_inbound_cidrs"></a> [allowed\_inbound\_cidrs](#input\_allowed\_inbound\_cidrs) | Which IPv4 addresses/ranges to allow access. This must be explicitly provided, and by default is set to ["*"] | `list(string)` | <pre>[<br>  "*"<br>]</pre> | no |
+| <a name="input_allowed_project_names"></a> [allowed\_project\_names](#input\_allowed\_project\_names) | A map of allowed projects where each key is a project number and the value is the connection limit. | `map(number)` | `{}` | no |
 | <a name="input_app_wandb_env"></a> [app\_wandb\_env](#input\_app\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Use an existing bucket. | `string` | `""` | no |
+| <a name="input_create_private_link"></a> [create\_private\_link](#input\_create\_private\_link) | Whether to create a private link service. | `bool` | `false` | no |
 | <a name="input_create_redis"></a> [create\_redis](#input\_create\_redis) | Boolean indicating whether to provision an redis instance (true) or not (false). | `bool` | `false` | no |
 | <a name="input_create_workload_identity"></a> [create\_workload\_identity](#input\_create\_workload\_identity) | Flag to indicate whether to create a workload identity for the service account. | `bool` | `true` | no |
 | <a name="input_database_machine_type"></a> [database\_machine\_type](#input\_database\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"db-n1-standard-2"` | no |
@@ -112,6 +118,7 @@ resources that lack official modules.
 | <a name="input_force_ssl"></a> [force\_ssl](#input\_force\_ssl) | Enforce SSL through the usage of the Cloud SQL Proxy (cloudsql://) in the DB connection string | `bool` | `false` | no |
 | <a name="input_gke_machine_type"></a> [gke\_machine\_type](#input\_gke\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"n1-standard-4"` | no |
 | <a name="input_gke_node_count"></a> [gke\_node\_count](#input\_gke\_node\_count) | n/a | `number` | `2` | no |
+| <a name="input_ilb_proxynetwork_cidr"></a> [ilb\_proxynetwork\_cidr](#input\_ilb\_proxynetwork\_cidr) | Internal load balancer proxy subnetwork | `string` | `"10.127.0.0/24"` | no |
 | <a name="input_kms_gcs_sa_name"></a> [kms\_gcs\_sa\_name](#input\_kms\_gcs\_sa\_name) | n/a | `string` | `"wandb-app"` | no |
 | <a name="input_labels"></a> [labels](#input\_labels) | Labels to apply to resources | `map(string)` | `{}` | no |
 | <a name="input_license"></a> [license](#input\_license) | Your wandb/local license | `string` | n/a | yes |
@@ -124,6 +131,8 @@ resources that lack official modules.
 | <a name="input_oidc_secret"></a> [oidc\_secret](#input\_oidc\_secret) | The Client secret of application in your identity provider | `string` | `""` | no |
 | <a name="input_other_wandb_env"></a> [other\_wandb\_env](#input\_other\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
 | <a name="input_parquet_wandb_env"></a> [parquet\_wandb\_env](#input\_parquet\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
+| <a name="input_psc_subnetwork_cidr"></a> [psc\_subnetwork\_cidr](#input\_psc\_subnetwork\_cidr) | Private link service reserved subnetwork | `string` | `"192.168.0.0/24"` | no |
+| <a name="input_public_access"></a> [public\_access](#input\_public\_access) | Whether to create a public endpoint for wandb access. | `bool` | `true` | no |
 | <a name="input_redis_reserved_ip_range"></a> [redis\_reserved\_ip\_range](#input\_redis\_reserved\_ip\_range) | Reserved IP range for REDIS peering connection | `string` | `"10.30.0.0/16"` | no |
 | <a name="input_redis_tier"></a> [redis\_tier](#input\_redis\_tier) | Specifies the tier for this Redis instance | `string` | `"STANDARD_HA"` | no |
 | <a name="input_resource_limits"></a> [resource\_limits](#input\_resource\_limits) | Specifies the resource limits for the wandb deployment | `map(string)` | <pre>{<br>  "cpu": null,<br>  "memory": null<br>}</pre> | no |
@@ -158,6 +167,7 @@ resources that lack official modules.
 | <a name="output_fqdn"></a> [fqdn](#output\_fqdn) | The FQDN to the W&B application |
 | <a name="output_gke_node_count"></a> [gke\_node\_count](#output\_gke\_node\_count) | n/a |
 | <a name="output_gke_node_instance_type"></a> [gke\_node\_instance\_type](#output\_gke\_node\_instance\_type) | n/a |
+| <a name="output_private_attachement_id"></a> [private\_attachement\_id](#output\_private\_attachement\_id) | n/a |
 | <a name="output_sa_account_email"></a> [sa\_account\_email](#output\_sa\_account\_email) | This output provides the email address of the service account created for workload identity, if workload identity is enabled. Otherwise, it returns null |
 | <a name="output_service_account"></a> [service\_account](#output\_service\_account) | Weights & Biases service account used to manage resources. |
 | <a name="output_standardized_size"></a> [standardized\_size](#output\_standardized\_size) | n/a |

main.tf

@@ -1,6 +1,6 @@
 module "project_factory_project_services" {
   source                      = "terraform-google-modules/project-factory/google//modules/project_services"
-  version                     = "~> 13.0"
+  version                     = "~> 14.0"
   project_id                  = null
   disable_dependent_services  = false
   disable_services_on_destroy = false
@@ -140,6 +140,14 @@ locals {
   secret_store_source     = "gcp-secretmanager://${local.project_id}?namespace=${var.namespace}"
 }
 
+resource "google_compute_address" "default" {
+  count        = var.create_private_link ? 1 : 0
+  name         = "${var.namespace}-ip-address"
+  subnetwork   = local.subnetwork.name
+  address_type = "INTERNAL"
+  purpose      = "GCE_ENDPOINT"
+}
+
 module "gke_app" {
   source  = "wandb/wandb/kubernetes"
   version = "1.14.1"
@@ -152,11 +160,11 @@ module "gke_app" {
   database_connection_string = module.database.connection_string
   redis_connection_string    = local.redis_connection_string
   redis_ca_cert              = local.redis_certificate
-  oidc_client_id   = var.oidc_client_id
-  oidc_issuer      = var.oidc_issuer
-  oidc_auth_method = var.oidc_auth_method
-  oidc_secret      = var.oidc_secret
-  local_restore    = var.local_restore
+  oidc_client_id             = var.oidc_client_id
+  oidc_issuer                = var.oidc_issuer
+  oidc_auth_method           = var.oidc_auth_method
+  oidc_secret                = var.oidc_secret
+  local_restore              = var.local_restore
 
   other_wandb_env = merge({
     "GORILLA_DISABLE_CODE_SAVING"          = var.disable_code_saving,
@@ -188,6 +196,7 @@ locals {
     "OIDC_AUTH_METHOD" = var.oidc_auth_method
     "OIDC_SECRET"      = var.oidc_secret
   } : {}
+  internal_lb_name = "${var.namespace}-internal"
 }
 
 data "google_client_config" "current" {}
@@ -257,13 +266,24 @@ module "wandb" {
       }
 
       ingress = {
+        create       = var.public_access # external ingress for public connection
         nameOverride = var.namespace
         annotations = {
           "kubernetes.io/ingress.class"                 = "gce"
           "kubernetes.io/ingress.global-static-ip-name" = module.app_lb.address_operator_name
           "ingress.gcp.kubernetes.io/pre-shared-cert"   = module.app_lb.certificate
         }
+        ## In order to support secondary ingress required min version 0.13.0 of operator-wandb chart
+        secondary = {
+          create       = var.create_private_link # internal ingress for private link connections
+          nameOverride = local.internal_lb_name
+          annotations = {
+            "kubernetes.io/ingress.class"                   = "gce-internal"
+            "kubernetes.io/ingress.regional-static-ip-name" = var.create_private_link ? google_compute_address.default[0].name : null
+          }
+        }
       }
+
       # To support otel rds and redis metrics need operator-wandb chart minimum version 0.13.8 ( stackdriver subchart)
       stackdriver = var.enable_stackdriver ? {
         install = true
@@ -338,3 +358,54 @@ module "wandb" {
     module.gke_app
   ]
 }
+
+# proxy-only subnet used by internal load balancer
+resource "google_compute_subnetwork" "proxy" {
+  name          = "${var.namespace}-proxy-subnet"
+  ip_cidr_range = var.ilb_proxynetwork_cidr
+  purpose       = "REGIONAL_MANAGED_PROXY"
+  role          = "ACTIVE"
+  network       = local.network.id
+}
+
+## This ensures that the private link resource does not fail during the provisioning process.
+module "sleep" {
+  source  = "matti/resource/shell"
+  version = "1.5.0"
+
+  environment = {
+    TIME = timestamp()
+  }
+  command              = "sleep 450; date +%s"
+  command_when_destroy = "sleep 450"
+  trigger              = timestamp()
+  working_dir          = "/tmp"
+
+  depends = [
+    module.wandb
+  ]
+}
+
+data "google_compute_forwarding_rules" "all" {
+  depends_on = [module.sleep.stdout]
+}
+
+locals {
+  regex_pattern       = local.internal_lb_name
+  filtered_rule_names = [for rule in data.google_compute_forwarding_rules.all.rules : rule.name if can(regex(local.regex_pattern, rule.name))]
+  forwarding_rule     = join(", ", local.filtered_rule_names)
+}
+
+## In order to support private link required min version 0.13.0 of operator-wandb chart
+module "private_link" {
+  count                 = var.create_private_link ? 1 : 0
+  source                = "./modules/private_link"
+  namespace             = var.namespace
+  forwarding_rule       = local.forwarding_rule
+  network               = local.network
+  subnetwork            = local.subnetwork
+  allowed_project_names = var.allowed_project_names
+  psc_subnetwork        = var.psc_subnetwork_cidr
+  proxynetwork_cidr     = var.ilb_proxynetwork_cidr
+  depends_on            = [google_compute_subnetwork.proxy, data.google_compute_forwarding_rules.all]
+}

modules/app_gke/outputs.tf

@@ -32,4 +32,6 @@ output "instance_group_url" {
 output "node_pool" {
   value = google_container_node_pool.default
 }
-
+output "mig_instance_group_id" {
+  value = google_container_node_pool.default.managed_instance_group_urls[0]
+}

modules/private_link/main.tf

@@ -0,0 +1,38 @@
+data "google_client_config" "current" {}
+
+resource "google_compute_service_attachment" "default" {
+  name                  = "${var.namespace}-private-link"
+  enable_proxy_protocol = false
+  connection_preference = "ACCEPT_MANUAL"
+  nat_subnets           = [google_compute_subnetwork.default.id]
+  target_service        = "https://www.googleapis.com/compute/v1/projects/${data.google_client_config.current.project}/regions/${data.google_client_config.current.region}/forwardingRules/${var.forwarding_rule}"
+
+  dynamic "consumer_accept_lists" {
+    for_each = var.allowed_project_names != {} ? var.allowed_project_names : {}
+    content {
+      project_id_or_num = consumer_accept_lists.key
+      connection_limit  = consumer_accept_lists.value
+    }
+  }
+  depends_on = [google_compute_subnetwork.default]
+}
+
+resource "google_compute_subnetwork" "default" {
+  name          = "${var.namespace}-psc-ilb-subnet"
+  network       = var.network.id
+  purpose       = "PRIVATE_SERVICE_CONNECT"
+  ip_cidr_range = var.psc_subnetwork
+}
+
+
+# allow all access from IAP and health check ranges
+resource "google_compute_firewall" "default" {
+  name          = "${var.namespace}-internal-fw"
+  provider      = google-beta
+  direction     = "INGRESS"
+  network       = var.network.id
+  source_ranges = ["130.211.0.0/22", "35.191.0.0/16", "35.235.240.0/20", var.proxynetwork_cidr]
+  allow {
+    protocol = "tcp"
+  }
+}

modules/private_link/outputs.tf

@@ -0,0 +1,3 @@
+output "private_attachement_id" {
+  value = try(google_compute_service_attachment.default.id,null)
+}

modules/private_link/variables.tf

@@ -0,0 +1,44 @@
+variable "namespace" {
+  type        = string
+  description = "The name prefix for all resources created."
+}
+
+variable "labels" {
+  description = "Labels which will be applied to all applicable resources."
+  type        = map(string)
+  default     = {}
+}
+
+variable "network" {
+  description = "Google Compute Engine network to which the cluster is connected."
+  type        = object({ id = string })
+}
+
+
+variable "subnetwork" {
+  type = object({
+    self_link = string
+  })
+  description = "The subnetwork object containing the self-link of the subnetwork."
+}
+
+variable "allowed_project_names" {
+  type = map(number)
+  default = {}
+  description = "A map of allowed projects where each key is a project number and the value is the connection limit."
+}
+
+variable "psc_subnetwork" {
+  type        = string
+  description = "Private link service reserved subnetwork"
+}
+
+variable "proxynetwork_cidr" {
+  type        = string
+  description = "Internal load balancer proxy subnetwork"
+}
+
+variable "forwarding_rule" {
+  type = string
+  description = "forwarding rule name used in private service connect as a target"
+}

outputs.tf

@@ -86,6 +86,10 @@ output "database_instance_type" {
   value = coalesce(try(local.deployment_size[var.size].db, null), var.database_machine_type)
 }
 
+output "private_attachement_id" {
+  value = var.create_private_link ? module.private_link[0].private_attachement_id : null
+}
+
 output "sa_account_email" {
   description =  "This output provides the email address of the service account created for workload identity, if workload identity is enabled. Otherwise, it returns null"
   value = var.create_workload_identity == true ? module.service_accounts.sa_account_email : null

variables.tf

@@ -254,6 +254,44 @@ variable "parquet_wandb_env" {
   default     = {}
 }
 
+##########################################
+# private link                           #
+##########################################
+
+## In order to support private link required min version 0.13.0 of operator-wandb chart
+
+variable "create_private_link" {
+  type        = bool
+  description = "Whether to create a private link service."
+  default     = false
+}
+
+variable "public_access" {
+  type        = bool
+  description = "Whether to create a public endpoint for wandb access."
+  default = true
+}
+
+variable "allowed_project_names" {
+  type = map(number)
+  default = {
+    # "project_ID" = 4 
+  }
+  description = "A map of allowed projects where each key is a project number and the value is the connection limit."
+}
+
+variable "psc_subnetwork_cidr" {
+  default     = "192.168.0.0/24"
+  description = "Private link service reserved subnetwork"
+  type        = string
+}
+
+variable "ilb_proxynetwork_cidr" {
+  default     = "10.127.0.0/24"
+  description = "Internal load balancer proxy subnetwork"
+  type        = string
+}
+
 variable "create_workload_identity" {
   description = "Flag to indicate whether to create a workload identity for the service account."
   type        = bool

versions.tf

@@ -1,9 +1,13 @@
 terraform {
   required_version = "~> 1.0"
   required_providers {
+    time = {
+      source = "hashicorp/time"
+      version = "0.11.2"
+    }
     google = {
       source  = "hashicorp/google"
-      version = "~> 4.82"
+      version = "~> 5.30"
     }
     kubernetes = {
       source  = "hashicorp/kubernetes"