Terraform module for provisioning and configuring google cloud composer instance and configure environment variables and install python packages.
-
Install and configure Terraform. You can follow the steps in following URL to install Terraform
-
Create a service account for creating cloud composer instance with cloud composer worker IAM role.
-
Update the path to service account in terraform main file.
Variable | Type | Comment |
---|---|---|
google_project_id | string | GCP project name |
name | string | Cloud composer instance name |
region | string | GCP compute node region |
zone | string | GCP compute zone |
machine_type | string | GCP node machine type |
string | Airflow alert mail address | |
sendgrid_api_key | string | API key for sendgrid |
oauth_scopes | list | oauth_scopes |
airflow_config_overrides | map | airflow_config_overrides |
pypi_packages | map | pypi_packages |
- You can a use the following snippet to create the cloud composer environment.
provider "google" {
credentials = "${file("account.json")}"
project = "dl-redash"
zone = "us-central1-c"
}
module "composer" {
source = "github.com/bmv3cg/cloud-composer"
google_project_id = "replace with your GCP project name"
name = "cloud-composer-tf"
zone= "us-central1-c"
region = "us-central1"
machine_type = "n1-standard-1"
sendgrid_api_key = "api key"
}
- You can intiate a Terrafrom module and provider using the command. This will download and configure all the dependencies for installation.
terraform init
- Customize the template as per your requirements. After confirming the plan you can run the apply the plan using the following execute the deployment.
terrafrom apply
Cloud composer provisioning takes around 10 to 15 minutes to complete, but if you are using custom environment changes and python modules it will nearly take 45 minutes to complete the deployment.