Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Improve BYOB Readme #302

Merged
merged 2 commits into from
Oct 16, 2024
Merged

fix: Improve BYOB Readme #302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a2c6f3e
improved the Readmes to point to our public documentations
mkaesz-wandb Oct 10, 2024
d3bfdb8
improved the Readmes to point to our public documentations
mkaesz-wandb Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 4 additions & 58 deletions examples/byob-sse-s3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,14 @@

## About

This example does not deploy an instance of Weights & Biases. Instead it is an
example of the resources that need to be created to deploy use with an S3 bucket
for.
Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector].

This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket and permissions) in the customer's account and grants the W&B AWS account access to them.

This module uses AE256 Encryption to protect the object store.

---

When using bring your own bucket you will need to grant our account
(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption.
decryption

## Using Terraform

Terraform is the preferred method for deploying BYOB.
Expand Down Expand Up @@ -59,54 +55,4 @@ Do not configure a KMS key on the object store. Your configuration should look l

### Creating S3 Bucket

Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
```

As stated above, server side encryption will be handled via SSE-S3 encryption with AE256.

Finally, grant the Weights & Biases Deployment account access to this S3 bucket:

```json
{
"Version": "2012-10-17",
"Id": "WandBAccess",
"Statement": [
{
"Sid": "WAndBAccountAccess",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::830241207209:root" },
"Action": [
"s3:GetObject*",
"s3:GetEncryptionConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::<WANDB_BUCKET>",
"arn:aws:s3:::<WANDB_BUCKET>/*"
]
}
]
}
```
Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually.
102 changes: 4 additions & 98 deletions examples/byob/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,11 @@

## About

This example does not deploy an instance of Weights & Biases. Instead it is an
example of the resources that need to be created to deploy use with an S3 bucket
for.
Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector].

---
This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket, KMS key and permissions) in the customer's account and grants the W&B AWS account access to them.

When using bring your own bucket you will need to grant our account
(`830241207209`) access to an S3 Bucket and KMS Key for encryption and decryption.
decryption
---

## Using Terraform

Expand Down Expand Up @@ -40,94 +36,4 @@ can version, reuse, and share.

## Using AWS Console

### Creating KMS Key

We require you to provision a KMS Key which will be used to encrypt and decrypt
your S3 bucket. Make sure to enable key usage type for `ENCRYPT_DECRYPT`
purposes. It will require to have the following policy:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid" : "Internal",
"Effect" : "Allow",
"Principal" : { "AWS" : "<you account id>" },
"Action" : "kms:*",
"Resource" : "<aws_kms_key.key.arn>"
},
{
"Sid" : "External",
"Effect" : "Allow",
"Principal" : { "AWS" : "arn:aws:iam::830241207209:root" },
"Action" : [
"kms:Decrypt",
"kms:Describe*",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*"
],
"Resource" : "<aws_kms_key.key.arn>"
}
]
}
```

This policy gives access to your internal account, a swell while also providing
our service account with the requires permissions. Please keep a record of the
KMS ARN as we will need that during the deployment.

### Creating S3 Bucket

Lastly, you'll need to create the S3 bucket. Make sure to enable CORS access. Your CORS configuration should look like the following:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
```

Also, enable server side encryption and use the KMS key you just generated.

Finally, grant the Weights & Biases Deployment account access to this S3 bucket:

```json
{
"Version": "2012-10-17",
"Id": "WandBAccess",
"Statement": [
{
"Sid": "WAndBAccountAccess",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::830241207209:root" },
"Action" : [
"s3:GetObject*",
"s3:GetEncryptionConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject",
"s3:GetBucketCORS",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::<WANDB_BUCKET>",
"arn:aws:s3:::<WANDB_BUCKET>/*"
]
}
]
}
```
Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually.
31 changes: 21 additions & 10 deletions modules/secure_storage_connector/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
# Weights & Biases Secure Storage Connector Module

This is a Terraform module for provisioning an s3 bucket to be used with Weights and Biases.
A KMS key used to encrypt S3 objects will also be created by default. To use this bucket with Weights and Biases
multi-tenant cloud, pass `arn:aws:iam::725579432336:role/WandbIntegration` for the `aws_principal_arn` variable.
## About

## AWS Services Used
Weights & Biases can connect to a S3 bucket created and owned by the customer. This is called BYOB (Bring your own bucket). More details (here)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector].

This example does not deploy a Weights & Biases instance. It deploys all required resources (S3 bucket, KMS key and permissions) in the customer's account and grants the W&B AWS account access to them.

---

## Using Terraform

### AWS Services Used

- AWS Identity & Access Management (IAM)
- AWS Key Management System (KMS)
- Amazon S3

## How to Use This Module
### How to Use This Module

- Ensure account meets module pre-requisites from above.
- Create a Terraform configuration that pulls in this module and specifies
Expand All @@ -35,20 +41,20 @@ module "secure_storage_connector" {

<!-- BEGIN_TF_DOCS -->

## Requirements
### Requirements

| Name | Version |
| --------------------------------------------------------------------------- | ------- |
| <a name="requirement_terraform"></a> [terraform](#requirement_terraform) | ~> 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement_aws) | ~> 3.60 |

## Providers
### Providers

| Name | Version |
| ------------------------------------------------ | ------- |
| <a name="provider_aws"></a> [aws](#provider_aws) | 3.61.0 |

## Inputs
### Inputs

| Name | Description | Type | Default | Required |
|--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------|:--------:|
Expand All @@ -58,12 +64,17 @@ module "secure_storage_connector" {
| <a name="input_sse_algorithm"></a> [sse_algorithm](#input_sse_algorithm) | The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms` | `string` | `aws:kms` | no |
| <a name="input_aws_principal_arn"></a> [aws_principal_arn](#input_aws_principal_arn) | AWS principal that can access the bucket | `string` | `null` | yes |

## Outputs
### Outputs

| Name | Description |
|-----------------------------------------------------------------------------|-------------------------------------------------------------------------|
| <a name="bucket_name"></a> [bucket_name](#bucket_name) | The name of the bucket created |
| <a name="bucket_arn"></a> [bucket_arn](#output_bucket_arn) | The arn of the bucket created |
| <a name="bucket_kms_key_arn"></a> [bucket_kms_key_arn](#bucket_kms_key_arn) | The arn of the kms key created |

<!-- END_TF_DOCS -->
<!-- END_TF_DOCS -->


## Using AWS Console

Please refer to the (public documentation)[https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector#provision-the-kms-key] on how to create all required resources manually.
Loading