Modules and strategies for postgres upgrade

docs/migration-guide/guide.md

@@ -0,0 +1,165 @@
+# PostgreSQL Migration Documentation
+
+
+## Known Limitations
+Before proceeding, please review the [known limitations](https://cloud.google.com/database-migration/docs/postgres/known-limitations) of the Database Migration Service (DMS).
+
+# Step 1: Configure Source Instance
+1. Select the Instance to Upgrade:
+- Decide upon a instance to upgrade:
+
+	- We are choosing the rishi-pg14-volcano-staging-pg-a34e9984 instance, a PostgreSQL 14 instance managed via the `galoy-infra/modules/postgresql/gcp` Terraform module.
+
+  ![decide-source](./assets/decide-source-instance.png)
+
+	- The source instance needs to be configured as [follows](https://cloud.google.com/database-migration/docs/postgres/configure-source-database#configure-your-source-instance-postgres)
+
+    	- We use a conditional flag in the terraform `galoy-infra/modules/postgresql/gcp` [here](https://github.com/GaloyMoney/galoy-infra/pull/190) 
+
+
+# Step 2: Create connection profile:
+[**Connection Profile Reference**](https://cloud.google.com/database-migration/docs/postgres/create-source-connection-profile)
+
+- A connection profile is configured via the terraform module mentioned above when we enable the `upgradable` flag.
+
+
+# Step 3: Configure Connectivity
+[**VPC-Peering Reference**](https://cloud.google.com/database-migration/docs/postgres/configure-connectivity-vpc-peering)
+
+- For the connectivity we would be using the internal vpc that we already employ, so we don't need to do anything else.
+
+
+# Step 4: Configure Destination
+- Configure New PostgreSQL Instance:
+	- **NOTE**: For simplicity keep the **prefix name** of source and destination same. 
+	- Use this Terraform [module](https://github.com/k3yss/galoy-infra/tree/work/keys/expose-sql-credentials/examples/gcp/db_migration/pg15) for configuration.
+	- You can also use create a new instance via the Database migration tool, but I find it a little confusing and complicated.
+
+![Point-in-time](./assets/point-in-time.png)
+- This warning is probably irrevant for us, as we would be using terraform for the configuration. But something to look out **for**! 
+
+# Step 5: Start Database Migration Process 
+
+![step-1](./assets/step-1.png)
+![step-2](./assets/step-2.png)
+![step-3](./assets/step-3.png)
+![step-4](./assets/step-4.png)
+![step-5](./assets/step-5.png)
+![step-6](./assets/step-6.png)
+![step-7](./assets/step-7.png)
+![step-8](./assets/step-8.png)
+
+Once the replication delay is zero, promote the migration.
+
+![promote-migration](./assets/promote-migration.png)
+![migration-completed](./assets/migration-completed.png)
+
+The Migration was successful.
+
+![migration-successful](./assets/successful-migration.png)
+
+# Step 6: Post-Migration Steps
+
+- [Verify Migration Job](https://cloud.google.com/database-migration/docs/postgres/quickstart#verify_the_migration_job) 
+
+> Once you migrated the database using DMS all objects and schema owner will become ‘cloudsqlexternalsync’ by default.
+
+1. Reassign Schema and Object Owners:
+     - After migration, all objects and schema owners become cloudsqlexternalsync. Reassign the schema and object owners to match the source instance.
+2. Migrate Users and Privileges:
+   - Migration does not transfer privileges and users. Create users manually based on the old database.
+
+# Step 6.5: Terraform state sync and user creation
+
+## Method 1
+This method requires some downtime as we will promote the instance first. Once the migration is completed, promote your instance and follow the steps below to sync the Terraform state.
+
+### Step 1
+Log in to the destination instance as the `postgres` user and change the name of the `cloudsqlexternalsync` user to your source database admin name:
+
+```sql
+ALTER USER "cloudsqlexternalsync" RENAME TO "<admin-user-name>";
+```
+
+### Step 2
+Modify your `main.tf` to reflect the new destination instance by changing the `database_version`:
+
+```hcl
+variable "name_prefix" {}
+variable "gcp_project" {}
+variable "destroyable_postgres" {
+  default = true
+}
+
+module "postgresql" {
+  #source = "git::https://github.com/GaloyMoney/galoy-infra.git//modules/postgresql/gcp?ref=689daa7"
+  source = "../../../modules/postgresql/gcp"
+
+  instance_name          = "rishi-pg"
+  vpc_name               = "${var.name_prefix}-vpc"
+  gcp_project            = var.gcp_project
+  destroyable            = var.destroyable_postgres
+  user_can_create_db     = true
+  databases              = ["test"]
+  highly_available       = false
+  database_version       = "POSTGRES_15"
+  replication            = false
+  provision_read_replica = false
+  upgradable             = false
+}
+```
+
+### Step 3
+Remove the state of the old instance. Below is the list of states to remove when the source instance has only one database named `test`:
+
+```sh
+terraform state rm module.postgresql.module.database[\"test\"]
+terraform state rm module.postgresql.google_database_migration_service_connection_profile.connection_profile[0]
+terraform state rm module.postgresql.random_password.migration[0]
+terraform state rm module.postgresql.google_sql_database_instance.instance
+terraform state rm $(terraform state list | grep module.postgresql.postgresql_grant)
+terraform state rm module.postgresql.postgresql_extension.pglogical[0]
+terraform state rm module.postgresql.random_id.db_name_suffix
+terraform state rm module.postgresql.google_sql_user.admin
+```
+
+Final state:
+```sh
+terraform state list
+module.postgresql.data.google_compute_network.vpc
+module.postgresql.random_password.admin
+```
+
+### Step 4
+Create an `import.tf` file with the following content:
+
+```hcl
+import {
+  to = module.postgresql.random_id.db_name_suffix
+  id = "<b64_url of your db_name_suffix>"
+}
+
+import {
+  to = module.postgresql.google_sql_database_instance.instance
+  id = "projects/volcano-staging/instances/<instance-name>"
+}
+
+import {
+  to = module.postgresql.module.database["test"].postgresql_database.db
+  id = "test"
+}
+```
+
+> To generate `db_name_suffix`, run:
+> ```sh
+> echo "<db-suffix>" | xxd -r -p | base64 | tr '/+' '_-' | tr -d '='
+> ```
+
+### Step 3 
+
+Finally, do a 
+
+```sh
+tf apply
+```
+and the state should be synced with the new PostgreSQL instance.

examples/gcp/db_migration/connection_profile/main.tf

@@ -0,0 +1,37 @@
+variable "postgres_host" {}
+variable "postgres_user_name" {}
+variable "postgres_port" {}
+variable "postgres_password" {}
+variable "cloud_sql_id" {}
+
+resource "google_database_migration_service_connection_profile" "postgresprofile" {
+  location              = "us-east1"
+  connection_profile_id = "db-migration-connection-profile"
+  display_name          = "db-migration"
+  // don't know why I would need a label yet
+  labels = {
+    foo = "bar"
+  }
+  postgresql {
+    host         = var.postgres_host
+    port         = var.postgres_port
+    username     = var.postgres_user_name
+    password     = var.postgres_password
+    cloud_sql_id = var.cloud_sql_id
+  }
+}
+
+terraform {
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = "5.36.0"
+    }
+  }
+}
+
+provider "google" {
+  project = "volcano-staging"
+  region  = "us-east1"
+  # Configuration options
+}

examples/gcp/db_migration/import-pg15/main.tf

@@ -0,0 +1,9 @@
+# importing the provider first
+terraform {
+  required_providers {
+    postgresql = {
+      source  = "cyrilgdn/postgresql"
+      version = "1.22.0"
+    }
+  }
+}

examples/gcp/db_migration/pg14/main.tf

@@ -0,0 +1,40 @@
+variable "name_prefix" {}
+variable "gcp_project" {}
+variable "vpc_name_prefix" {}
+variable "database_version" {}
+variable "destroyable_postgres" {
+  default = false
+}
+
+output "instance_name" {
+  value = module.postgresql.instance_name
+}
+
+output "private_ip" {
+  value = module.postgresql.private_ip
+}
+
+output "instance_creds" {
+  value     = module.postgresql.instance_creds
+  sensitive = true
+}
+
+output "creds" {
+  value     = module.postgresql.creds
+  sensitive = true
+}
+
+module "postgresql" {
+  # source = "git::https://github.com/GaloyMoney/galoy-infra.git//modules/postgresql/gcp?ref=1eb536b"
+  source = "../../../../modules/postgresql/gcp"
+
+  instance_name          = "${var.name_prefix}-pg"
+  vpc_name               = "${var.vpc_name_prefix}-vpc"
+  gcp_project            = var.gcp_project
+  destroyable            = var.destroyable_postgres
+  user_can_create_db     = true
+  databases              = ["test", "test2"]
+  replication            = true
+  provision_read_replica = false
+  database_version       = var.database_version
+}

examples/gcp/db_migration/pg15/main.tf

@@ -0,0 +1,34 @@
+variable "name_prefix" {}
+variable "gcp_project" {}
+variable "vpc_name_prefix" {}
+variable "instance_admin_password" {}
+variable "database_version" {}
+variable "destroyable_postgres" {
+  default = false
+}
+
+output "instance_name" {
+  value = module.postgresql.instance_name
+}
+
+output "private_ip" {
+  value = module.postgresql.private_ip
+}
+
+output "instance_creds" {
+  value     = module.postgresql.instance_creds
+  sensitive = true
+}
+
+module "postgresql" {
+  # source = "git::https://github.com/GaloyMoney/galoy-infra.git//modules/postgresql/gcp?ref=1eb536b"
+  source = "../../../../modules/postgresql/gcp-pg15"
+
+  instance_name           = "${var.name_prefix}-pg"
+  vpc_name                = "${var.vpc_name_prefix}-vpc"
+  gcp_project             = var.gcp_project
+  instance_admin_password = var.instance_admin_password
+  destroyable             = var.destroyable_postgres
+  databases               = ["test"]
+  database_version        = var.database_version
+}

examples/gcp/postgresql/main.tf

@@ -1,19 +1,40 @@
 variable "name_prefix" {}
 variable "gcp_project" {}
+variable "vpc_name_prefix" {}
+variable "database_version" {}
 variable "destroyable_postgres" {
   default = false
 }
 
+output "instance_name" {
+  value = module.postgresql.instance_name
+}
+
+output "private_ip" {
+  value = module.postgresql.private_ip
+}
+
+output "instance_creds" {
+  value     = module.postgresql.instance_creds
+  sensitive = true
+}
+
+output "creds" {
+  value     = module.postgresql.creds
+  sensitive = true
+}
+
 module "postgresql" {
-  source = "git::https://github.com/GaloyMoney/galoy-infra.git//modules/postgresql/gcp?ref=1eb536b"
-  # source = "../../../modules/postgresql/gcp"
+  # source = "git::https://github.com/GaloyMoney/galoy-infra.git//modules/postgresql/gcp?ref=1eb536b"
+  source = "../../../modules/postgresql/gcp"
 
   instance_name          = "${var.name_prefix}-pg"
-  vpc_name               = "${var.name_prefix}-vpc"
+  vpc_name               = "${var.vpc_name_prefix}-vpc"
   gcp_project            = var.gcp_project
   destroyable            = var.destroyable_postgres
   user_can_create_db     = true
-  databases              = []
+  databases              = ["test"]
   replication            = true
-  provision_read_replica = true
+  provision_read_replica = false
+  database_version       = var.database_version
 }

modules/postgresql/gcp-pg15/main.tf

@@ -0,0 +1,59 @@
+data "google_compute_network" "vpc" {
+  project = local.gcp_project
+  name    = local.vpc_name
+}
+
+resource "random_id" "db_name_suffix" {
+  byte_length = 4
+}
+
+resource "google_sql_database_instance" "instance" {
+  name = "${local.instance_name}-${random_id.db_name_suffix.hex}"
+
+  project             = local.gcp_project
+  database_version    = local.database_version
+  region              = local.region
+  deletion_protection = !local.destroyable
+
+  settings {
+    tier              = local.tier
+    availability_type = local.highly_available ? "REGIONAL" : "ZONAL"
+
+    ip_configuration {
+      ipv4_enabled                                  = false
+      private_network                               = data.google_compute_network.vpc.id
+      enable_private_path_for_google_cloud_services = true
+    }
+  }
+
+  timeouts {
+    create = "45m"
+    update = "45m"
+    delete = "45m"
+  }
+}
+
+resource "google_sql_user" "admin" {
+  name     = "postgres"
+  instance = google_sql_database_instance.instance.name
+  password = local.instance_admin_password
+  project  = local.gcp_project
+}
+
+provider "postgresql" {
+  host     = google_sql_database_instance.instance.private_ip_address
+  username = google_sql_user.admin.name
+  password = local.instance_admin_password
+
+  # GCP doesn't let superuser mode https://cloud.google.com/sql/docs/postgres/users#superuser_restrictions
+  superuser = false
+}
+
+terraform {
+  required_providers {
+    postgresql = {
+      source  = "cyrilgdn/postgresql"
+      version = "1.22.0"
+    }
+  }
+}