Skip to content

Commit

Permalink
fix: SSE-S3 example (#132)
Browse files Browse the repository at this point in the history
* fix: SSE-S3 example

* docs: Update some sse refferences
  • Loading branch information
zacharyblasczyk authored Aug 30, 2023
1 parent 1ef5838 commit 627005b
Show file tree
Hide file tree
Showing 4 changed files with 175 additions and 2 deletions.
112 changes: 112 additions & 0 deletions examples/byob-sse-s3/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# BYOB

## About

This example does not deploy an instance of Weights & Biases. Instead it is an
example of the resources that need to be created to deploy use with an S3 bucket
for.

This module uses AE256 Encryption to protect the object store.

---

When using bring your own bucket you will need to grant our account
(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption.
decryption

## Using Terraform

Terraform is the preferred method for deploying BYOB.

Infrastructure as code (IaC) tools allow you to manage infrastructure with
configuration files rather than through a graphical user interface. IaC
allows you to build, change, and manage your infrastructure in a safe,
consistent, and repeatable way by defining resource configurations that you
can version, reuse, and share.

1. Please follow the instructions for install [Terraform
1.0+](https://learn.hashicorp.com/tutorials/terraform/install-cli)
2. Authenticated with Terraform with AWS. You can do this in many ways learn
more
[here](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration).
It is most common to install and authenticate with [AWS
CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html).
3. Pull terraform-aws-wandb repo and cd to this
[directory](https://github.com/wandb/terraform-aws-wandb/tree/main/examples/byob-sse-s3)
4. Run `terraform init`
5. Run `terraform apply`. If you need to assume a different role, please
configure that in the `main.tf` file before running `apply`. You can learn
more
[here](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#assuming-an-iam-role).
6. Please provide the resulting output to Weights & Biases (bucket name and kms arn)

## Using AWS Console

### SSE-S3 encryption

Amazon S3 now applies server-side encryption with Amazon S3 managed keys (SSE-S3)
as the base level of encryption for every bucket in Amazon S3. Starting January 5, 2023,
all new object uploads to Amazon S3 are automatically encrypted at no additional cost
and with no impact on performance. The automatic encryption status for S3 bucket default
encryption configuration and for new object uploads is available in AWS CloudTrail logs,
S3 Inventory, S3 Storage Lens, the Amazon S3 console, and as an additional Amazon S3 API
response header in the AWS Command Line Interface and AWS SDKs. For more information, see
[Default encryption FAQ](https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-encryption-faq.html).

Do not configure a KMS key on the object store. Your configuration should look like this.

![sse-s3-default](./sse-s3.png)

### Creating S3 Bucket

Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
```

As stated above, server side encryption will be handled via SSE-S3 encryption with AE256.

Finally, grant the Weights & Biases Deployment account access to this S3 bucket:

```json
{
"Version": "2012-10-17",
"Id": "WandBAccess",
"Statement": [
{
"Sid": "WAndBAccountAccess",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::830241207209:root" },
"Action": [
"s3:GetObject*",
"s3:GetEncryptionConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::<WANDB_BUCKET>",
"arn:aws:s3:::<WANDB_BUCKET>/*"
]
}
]
}
```
61 changes: 61 additions & 0 deletions examples/byob-sse-s3/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@

variable "bucket_prefix" {
type = string
description = "Prefix of your bucket"
}

variable "region" {
type = string
description = "AWS region the bucket will live in."
}

variable "eks_node_role_arn" {
type = string
description = "EKS node role for cross account access."
default = ""
}

variable "sse_algorithm" {
description = "The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms`"
type = string
default = "AES256"
}

variable "create_kms_key" {
description = "If a KMS key should be created to encrypt S3 storage bucket objects. This can only be used when you set the value of sse_algorithm as aws:kms."
type = bool
default = false
}

provider "aws" {
region = var.region

default_tags {
tags = {
GithubRepo = "terraform-aws-wandb"
GithubOrg = "wandb"
Enviroment = "BringYourOwnBucket"
Namespace = "WeightsBiases"
}
}
}

locals {
namespace = var.bucket_prefix

# Weights & Biases Deployment Account
wandb_deployment_account_id = "830241207209"
wandb_deployment_account_arn = var.eks_node_role_arn == "" ? "arn:aws:iam::${local.wandb_deployment_account_id}:root" : var.eks_node_role_arn
}

module "secure_storage_connector" {
source = "wandb/wandb/aws//modules/secure_storage_connector"
namespace = local.namespace
aws_principal_arn = local.wandb_deployment_account_arn
sse_algorithm = var.sse_algorithm
create_kms_key = var.create_kms_key
}

output "bucket_name" {
value = module.secure_storage_connector.bucket.bucket
}
Binary file added examples/byob-sse-s3/sse-s3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions modules/secure_storage_connector/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ module "file_storage" {
source = "../../modules/file_storage"

namespace = var.namespace
sse_algorithm = "aws:kms"
sse_algorithm = var.sse_algorithm
kms_key_arn = var.create_kms_key ? aws_kms_key.key[0].arn : null

create_queue = false
Expand Down Expand Up @@ -80,4 +80,4 @@ resource "aws_s3_bucket_policy" "s3_policy" {

data "aws_s3_bucket" "file_storage" {
bucket = module.file_storage.bucket_name
}
}

0 comments on commit 627005b

Please sign in to comment.