From 47975f17a9205b614f9e24f64ac136cd7f20bc6a Mon Sep 17 00:00:00 2001 From: mkaesz-wandb <165807238+mkaesz-wandb@users.noreply.github.com> Date: Wed, 16 Oct 2024 15:30:54 +0200 Subject: [PATCH] fix: Improve BYOB Readme (#302) * improved the Readmes to point to our public documentations * improved the Readmes to point to our public documentations --- examples/byob-sse-s3/README.md | 62 +------------ examples/byob/README.md | 102 +-------------------- modules/secure_storage_connector/README.md | 31 +++++-- 3 files changed, 29 insertions(+), 166 deletions(-) diff --git a/examples/byob-sse-s3/README.md b/examples/byob-sse-s3/README.md index 6aa9fb36..e5f81fde 100644 --- a/examples/byob-sse-s3/README.md +++ b/examples/byob-sse-s3/README.md @@ -2,18 +2,14 @@ ## About -This example does not deploy an instance of Weights & Biases. Instead it is an -example of the resources that need to be created to deploy use with an S3 bucket -for. +Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector]. + +This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket and permissions) in the customer's account and grants the W&B AWS account access to them. This module uses AE256 Encryption to protect the object store. --- -When using bring your own bucket you will need to grant our account -(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption. -decryption - ## Using Terraform Terraform is the preferred method for deploying BYOB. @@ -59,54 +55,4 @@ Do not configure a KMS key on the object store. Your configuration should look l ### Creating S3 Bucket -Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following: - -```xml - - - - * - GET - HEAD - PUT - * - ETag - 3000 - - -``` - -As stated above, server side encryption will be handled via SSE-S3 encryption with AE256. - -Finally, grant the Weights & Biases Deployment account access to this S3 bucket: - -```json -{ - "Version": "2012-10-17", - "Id": "WandBAccess", - "Statement": [ - { - "Sid": "WAndBAccountAccess", - "Effect": "Allow", - "Principal": { "AWS": "arn:aws:iam::830241207209:root" }, - "Action": [ - "s3:GetObject*", - "s3:GetEncryptionConfiguration", - "s3:ListBucket", - "s3:ListBucketMultipartUploads", - "s3:ListBucketVersions", - "s3:AbortMultipartUpload", - "s3:DeleteObject", - "s3:PutObject", - "s3:GetBucketCORS", - "s3:GetBucketLocation", - "s3:GetBucketVersioning" - ], - "Resource": [ - "arn:aws:s3:::", - "arn:aws:s3:::/*" - ] - } - ] -} -``` +Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually. \ No newline at end of file diff --git a/examples/byob/README.md b/examples/byob/README.md index e4d8589d..be9854ba 100644 --- a/examples/byob/README.md +++ b/examples/byob/README.md @@ -2,15 +2,11 @@ ## About -This example does not deploy an instance of Weights & Biases. Instead it is an -example of the resources that need to be created to deploy use with an S3 bucket -for. +Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector]. ---- +This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket, KMS key and permissions) in the customer's account and grants the W&B AWS account access to them. -When using bring your own bucket you will need to grant our account -(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption. -decryption +--- ## Using Terraform @@ -40,94 +36,4 @@ can version, reuse, and share. ## Using AWS Console -### Creating KMS Key - -We require you to provision a KMS Key which will be used to encrypt and decrypt -your S3 bucket. Make sure to enable key usage type for `ENCRYPT_DECRYPT` -purposes. It will require to have the following policy: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid" : "Internal", - "Effect" : "Allow", - "Principal" : { "AWS" : "" }, - "Action" : "kms:*", - "Resource" : "" - }, - { - "Sid" : "External", - "Effect" : "Allow", - "Principal" : { "AWS" : "arn:aws:iam::830241207209:root" }, - "Action" : [ - "kms:Decrypt", - "kms:Describe*", - "kms:Encrypt", - "kms:ReEncrypt*", - "kms:GenerateDataKey*" - ], - "Resource" : "" - } - ] -} -``` - -This policy gives access to your internal account, a swell while also providing -our service account with the requires permissions. Please keep a record of the -KMS ARN as we will need that during the deployment. - -### Creating S3 Bucket - -Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following: - -```xml - - - - * - GET - HEAD - PUT - * - ETag - 3000 - - -``` - -Also, enable server side encryption and use the KMS key you just generated. - -Finally, grant the Weights & Biases Deployment account access to this S3 bucket: - -```json -{ - "Version": "2012-10-17", - "Id": "WandBAccess", - "Statement": [ - { - "Sid": "WAndBAccountAccess", - "Effect": "Allow", - "Principal": { "AWS": "arn:aws:iam::830241207209:root" }, - "Action" : [ - "s3:GetObject*", - "s3:GetEncryptionConfiguration", - "s3:ListBucket", - "s3:ListBucketMultipartUploads", - "s3:ListBucketVersions", - "s3:AbortMultipartUpload", - "s3:DeleteObject", - "s3:PutObject", - "s3:GetBucketCORS", - "s3:GetBucketLocation", - "s3:GetBucketVersioning" - ], - "Resource": [ - "arn:aws:s3:::", - "arn:aws:s3:::/*" - ] - } - ] -} -``` +Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually. diff --git a/modules/secure_storage_connector/README.md b/modules/secure_storage_connector/README.md index b2bdee00..27382705 100644 --- a/modules/secure_storage_connector/README.md +++ b/modules/secure_storage_connector/README.md @@ -1,16 +1,22 @@ # Weights & Biases Secure Storage Connector Module -This is a Terraform module for provisioning an s3 bucket to be used with Weights and Biases. -A KMS key used to encrypt S3 objects will also be created by default. To use this bucket with Weights and Biases -multi-tenant cloud, pass `arn:aws:iam::725579432336:role/WandbIntegration` for the `aws_principal_arn` variable. +## About -## AWS Services Used +Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector]. + +This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket, KMS key and permissions) in the customer's account and grants the W&B AWS account access to them. + +--- + +## Using Terraform + +### AWS Services Used - AWS Identity & Access Management (IAM) - AWS Key Management System (KMS) - Amazon S3 -## How to Use This Module +### How to Use This Module - Ensure account meets module pre-requisites from above. - Create a Terraform configuration that pulls in this module and specifies @@ -35,20 +41,20 @@ module "secure_storage_connector" { -## Requirements +### Requirements | Name | Version | | --------------------------------------------------------------------------- | ------- | | [terraform](#requirement_terraform) | ~> 1.0 | | [aws](#requirement_aws) | ~> 3.60 | -## Providers +### Providers | Name | Version | | ------------------------------------------------ | ------- | | [aws](#provider_aws) | 3.61.0 | -## Inputs +### Inputs | Name | Description | Type | Default | Required | |--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------|:--------:| @@ -58,7 +64,7 @@ module "secure_storage_connector" { | [sse_algorithm](#input_sse_algorithm) | The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms` | `string` | `aws:kms` | no | | [aws_principal_arn](#input_aws_principal_arn) | AWS principal that can access the bucket | `string` | `null` | yes | -## Outputs +### Outputs | Name | Description | |-----------------------------------------------------------------------------|-------------------------------------------------------------------------| @@ -66,4 +72,9 @@ module "secure_storage_connector" { | [bucket_arn](#output_bucket_arn) | The arn of the bucket created | | [bucket_kms_key_arn](#bucket_kms_key_arn) | The arn of the kms key created | - \ No newline at end of file + + + +## Using AWS Console + +Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually. \ No newline at end of file