-
Notifications
You must be signed in to change notification settings - Fork 19
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* fix: SSE-S3 example * docs: Update some sse refferences
- Loading branch information
1 parent
1ef5838
commit 627005b
Showing
4 changed files
with
175 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
# BYOB | ||
|
||
## About | ||
|
||
This example does not deploy an instance of Weights & Biases. Instead it is an | ||
example of the resources that need to be created to deploy use with an S3 bucket | ||
for. | ||
|
||
This module uses AE256 Encryption to protect the object store. | ||
|
||
--- | ||
|
||
When using bring your own bucket you will need to grant our account | ||
(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption. | ||
decryption | ||
|
||
## Using Terraform | ||
|
||
Terraform is the preferred method for deploying BYOB. | ||
|
||
Infrastructure as code (IaC) tools allow you to manage infrastructure with | ||
configuration files rather than through a graphical user interface. IaC | ||
allows you to build, change, and manage your infrastructure in a safe, | ||
consistent, and repeatable way by defining resource configurations that you | ||
can version, reuse, and share. | ||
|
||
1. Please follow the instructions for install [Terraform | ||
1.0+](https://learn.hashicorp.com/tutorials/terraform/install-cli) | ||
2. Authenticated with Terraform with AWS. You can do this in many ways learn | ||
more | ||
[here](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration). | ||
It is most common to install and authenticate with [AWS | ||
CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html). | ||
3. Pull terraform-aws-wandb repo and cd to this | ||
[directory](https://github.com/wandb/terraform-aws-wandb/tree/main/examples/byob-sse-s3) | ||
4. Run `terraform init` | ||
5. Run `terraform apply`. If you need to assume a different role, please | ||
configure that in the `main.tf` file before running `apply`. You can learn | ||
more | ||
[here](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#assuming-an-iam-role). | ||
6. Please provide the resulting output to Weights & Biases (bucket name and kms arn) | ||
|
||
## Using AWS Console | ||
|
||
### SSE-S3 encryption | ||
|
||
Amazon S3 now applies server-side encryption with Amazon S3 managed keys (SSE-S3) | ||
as the base level of encryption for every bucket in Amazon S3. Starting January 5, 2023, | ||
all new object uploads to Amazon S3 are automatically encrypted at no additional cost | ||
and with no impact on performance. The automatic encryption status for S3 bucket default | ||
encryption configuration and for new object uploads is available in AWS CloudTrail logs, | ||
S3 Inventory, S3 Storage Lens, the Amazon S3 console, and as an additional Amazon S3 API | ||
response header in the AWS Command Line Interface and AWS SDKs. For more information, see | ||
[Default encryption FAQ](https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-encryption-faq.html). | ||
|
||
Do not configure a KMS key on the object store. Your configuration should look like this. | ||
|
||
![sse-s3-default](./sse-s3.png) | ||
|
||
### Creating S3 Bucket | ||
|
||
Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following: | ||
|
||
```xml | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> | ||
<CORSRule> | ||
<AllowedOrigin>*</AllowedOrigin> | ||
<AllowedMethod>GET</AllowedMethod> | ||
<AllowedMethod>HEAD</AllowedMethod> | ||
<AllowedMethod>PUT</AllowedMethod> | ||
<AllowedHeader>*</AllowedHeader> | ||
<ExposeHeader>ETag</ExposeHeader> | ||
<MaxAgeSeconds>3000</MaxAgeSeconds> | ||
</CORSRule> | ||
</CORSConfiguration> | ||
``` | ||
|
||
As stated above, server side encryption will be handled via SSE-S3 encryption with AE256. | ||
|
||
Finally, grant the Weights & Biases Deployment account access to this S3 bucket: | ||
|
||
```json | ||
{ | ||
"Version": "2012-10-17", | ||
"Id": "WandBAccess", | ||
"Statement": [ | ||
{ | ||
"Sid": "WAndBAccountAccess", | ||
"Effect": "Allow", | ||
"Principal": { "AWS": "arn:aws:iam::830241207209:root" }, | ||
"Action": [ | ||
"s3:GetObject*", | ||
"s3:GetEncryptionConfiguration", | ||
"s3:ListBucket", | ||
"s3:ListBucketMultipartUploads", | ||
"s3:ListBucketVersions", | ||
"s3:AbortMultipartUpload", | ||
"s3:DeleteObject", | ||
"s3:PutObject", | ||
"s3:GetBucketCORS", | ||
"s3:GetBucketLocation", | ||
"s3:GetBucketVersioning" | ||
], | ||
"Resource": [ | ||
"arn:aws:s3:::<WANDB_BUCKET>", | ||
"arn:aws:s3:::<WANDB_BUCKET>/*" | ||
] | ||
} | ||
] | ||
} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
|
||
variable "bucket_prefix" { | ||
type = string | ||
description = "Prefix of your bucket" | ||
} | ||
|
||
variable "region" { | ||
type = string | ||
description = "AWS region the bucket will live in." | ||
} | ||
|
||
variable "eks_node_role_arn" { | ||
type = string | ||
description = "EKS node role for cross account access." | ||
default = "" | ||
} | ||
|
||
variable "sse_algorithm" { | ||
description = "The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms`" | ||
type = string | ||
default = "AES256" | ||
} | ||
|
||
variable "create_kms_key" { | ||
description = "If a KMS key should be created to encrypt S3 storage bucket objects. This can only be used when you set the value of sse_algorithm as aws:kms." | ||
type = bool | ||
default = false | ||
} | ||
|
||
provider "aws" { | ||
region = var.region | ||
|
||
default_tags { | ||
tags = { | ||
GithubRepo = "terraform-aws-wandb" | ||
GithubOrg = "wandb" | ||
Enviroment = "BringYourOwnBucket" | ||
Namespace = "WeightsBiases" | ||
} | ||
} | ||
} | ||
|
||
locals { | ||
namespace = var.bucket_prefix | ||
|
||
# Weights & Biases Deployment Account | ||
wandb_deployment_account_id = "830241207209" | ||
wandb_deployment_account_arn = var.eks_node_role_arn == "" ? "arn:aws:iam::${local.wandb_deployment_account_id}:root" : var.eks_node_role_arn | ||
} | ||
|
||
module "secure_storage_connector" { | ||
source = "wandb/wandb/aws//modules/secure_storage_connector" | ||
namespace = local.namespace | ||
aws_principal_arn = local.wandb_deployment_account_arn | ||
sse_algorithm = var.sse_algorithm | ||
create_kms_key = var.create_kms_key | ||
} | ||
|
||
output "bucket_name" { | ||
value = module.secure_storage_connector.bucket.bucket | ||
} |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters