-
Notifications
You must be signed in to change notification settings - Fork 27
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add/Modify OSI options on Migration Console (#643)
This change makes the following minor modifications for our OSI migration setup for the migration console *Removes incorrect separate OSI pipeline IAM roles for source and target clusters. This must be the same role and have adjusted accordingly *Add option to include-index-regex. This allows a user to specify multiple selection regexes that we will use to select which indices to migrate when configuring the template for OSI. As a default we will still try to move everything except system indices if not specified. *Properly hookup missing CW log settings following OSI required conventions *Minor cleanup in spots of OSI script Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
- Loading branch information
Showing
13 changed files
with
458 additions
and
73 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
50 changes: 50 additions & 0 deletions
50
TrafficCapture/dockerSolution/src/main/docker/migrationConsole/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
# Migration Console | ||
The accessible control hub for all things migrations | ||
|
||
|
||
## Running Python tests | ||
|
||
|
||
### Installing Requirements | ||
|
||
To isolate the Python environment for the project from your local machine, create virtual environment like so: | ||
``` | ||
python3 -m venv .venv | ||
source .venv/bin/activate | ||
``` | ||
|
||
You can exit the Python virtual environment and remove its resources like so: | ||
``` | ||
deactivate | ||
rm -rf .venv | ||
``` | ||
|
||
Install developer requirements for osiMigration library like so: | ||
``` | ||
pip install -r lib/osiMigrationLib/dev-requirements.txt | ||
``` | ||
|
||
|
||
### Unit Tests | ||
|
||
Unit tests can be run from this current `migrationConsole/` directory using: | ||
|
||
```shell | ||
python -m unittest | ||
``` | ||
|
||
### Coverage | ||
|
||
_Code coverage_ metrics can be generated after a unit-test run. A report can either be printed on the command line: | ||
|
||
```shell | ||
python -m coverage report --omit "*/test/*" | ||
``` | ||
|
||
or generated as HTML: | ||
|
||
```shell | ||
python -m coverage html --omit "*/test/*" | ||
``` | ||
|
||
Note that the `--omit` parameter must be specified to avoid tracking code coverage on unit test code itself. |
Empty file.
2 changes: 2 additions & 0 deletions
2
.../dockerSolution/src/main/docker/migrationConsole/lib/osiMigrationLib/dev-requirements.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
-r requirements.txt | ||
coverage |
189 changes: 131 additions & 58 deletions
189
TrafficCapture/dockerSolution/src/main/docker/migrationConsole/osiMigration.py
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
38 changes: 38 additions & 0 deletions
38
...lution/src/main/docker/migrationConsole/test/resources/basicAuthSourceAndSigv4Target.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
version: "2" | ||
# NOTE: Placeholder values will be automatically populated and do not need to be changed | ||
pipeline_configurations: | ||
aws: | ||
secrets: | ||
source-secret-config: | ||
secret_id: unit-test-secret | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 | ||
|
||
historical-data-migration: | ||
|
||
# Source cluster configuration | ||
source: | ||
opensearch: | ||
hosts: | ||
- https://vpc-test-123.com | ||
indices: | ||
# As a default this will be populated to move all indices except system indices (those that start with '.'), | ||
# but tool also allows passing regex strings of indices to include, in which case all indices matching the regex | ||
# will be migrated | ||
exclude: | ||
- index_name_regex: \.* | ||
username: "${{aws_secrets:source-secret-config:username}}" | ||
password: "${{aws_secrets:source-secret-config:password}}" | ||
|
||
# Target cluster configuration | ||
sink: | ||
- opensearch: | ||
hosts: | ||
- https://vpc-test-456.com | ||
# Derive index name from record metadata | ||
index: ${getMetadata("opensearch-index")} | ||
# Use the same document ID as the source cluster document | ||
document_id: ${getMetadata("opensearch-document_id")} | ||
aws: | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 |
40 changes: 40 additions & 0 deletions
40
...docker/migrationConsole/test/resources/basicAuthSourceWithMultipleIndexInclusionRule.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
version: "2" | ||
# NOTE: Placeholder values will be automatically populated and do not need to be changed | ||
pipeline_configurations: | ||
aws: | ||
secrets: | ||
source-secret-config: | ||
secret_id: unit-test-secret | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 | ||
|
||
historical-data-migration: | ||
|
||
# Source cluster configuration | ||
source: | ||
opensearch: | ||
hosts: | ||
- https://vpc-test-123.com | ||
indices: | ||
# As a default this will be populated to move all indices except system indices (those that start with '.'), | ||
# but tool also allows passing regex strings of indices to include, in which case all indices matching the regex | ||
# will be migrated | ||
include: | ||
- index_name_regex: index* | ||
- index_name_regex: .* | ||
- index_name_regex: sam[a-z]+ | ||
username: "${{aws_secrets:source-secret-config:username}}" | ||
password: "${{aws_secrets:source-secret-config:password}}" | ||
|
||
# Target cluster configuration | ||
sink: | ||
- opensearch: | ||
hosts: | ||
- https://vpc-test-456.com | ||
# Derive index name from record metadata | ||
index: ${getMetadata("opensearch-index")} | ||
# Use the same document ID as the source cluster document | ||
document_id: ${getMetadata("opensearch-document_id")} | ||
aws: | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 |
38 changes: 38 additions & 0 deletions
38
...n/docker/migrationConsole/test/resources/basicAuthSourceWithSingleIndexInclusionRule.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
version: "2" | ||
# NOTE: Placeholder values will be automatically populated and do not need to be changed | ||
pipeline_configurations: | ||
aws: | ||
secrets: | ||
source-secret-config: | ||
secret_id: unit-test-secret | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 | ||
|
||
historical-data-migration: | ||
|
||
# Source cluster configuration | ||
source: | ||
opensearch: | ||
hosts: | ||
- https://vpc-test-123.com | ||
indices: | ||
# As a default this will be populated to move all indices except system indices (those that start with '.'), | ||
# but tool also allows passing regex strings of indices to include, in which case all indices matching the regex | ||
# will be migrated | ||
include: | ||
- index_name_regex: index* | ||
username: "${{aws_secrets:source-secret-config:username}}" | ||
password: "${{aws_secrets:source-secret-config:password}}" | ||
|
||
# Target cluster configuration | ||
sink: | ||
- opensearch: | ||
hosts: | ||
- https://vpc-test-456.com | ||
# Derive index name from record metadata | ||
index: ${getMetadata("opensearch-index")} | ||
# Use the same document ID as the source cluster document | ||
document_id: ${getMetadata("opensearch-document_id")} | ||
aws: | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 |
33 changes: 33 additions & 0 deletions
33
...erSolution/src/main/docker/migrationConsole/test/resources/sigv4SourceAndSigv4Target.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
version: "2" | ||
# NOTE: Placeholder values will be automatically populated and do not need to be changed | ||
|
||
|
||
historical-data-migration: | ||
|
||
# Source cluster configuration | ||
source: | ||
opensearch: | ||
hosts: | ||
- https://vpc-test-123.com | ||
indices: | ||
# As a default this will be populated to move all indices except system indices (those that start with '.'), | ||
# but tool also allows passing regex strings of indices to include, in which case all indices matching the regex | ||
# will be migrated | ||
exclude: | ||
- index_name_regex: \.* | ||
aws: | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 | ||
|
||
# Target cluster configuration | ||
sink: | ||
- opensearch: | ||
hosts: | ||
- https://vpc-test-456.com | ||
# Derive index name from record metadata | ||
index: ${getMetadata("opensearch-index")} | ||
# Use the same document ID as the source cluster document | ||
document_id: ${getMetadata("opensearch-document_id")} | ||
aws: | ||
region: us-west-2 | ||
sts_role_arn: arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123 |
97 changes: 97 additions & 0 deletions
97
TrafficCapture/dockerSolution/src/main/docker/migrationConsole/test/testOSIMigration.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
import unittest | ||
|
||
import osiMigration | ||
|
||
# These values should map to static template values in the resources directory | ||
SOURCE_ENDPOINT = 'https://vpc-test-123.com' | ||
TARGET_ENDPOINT = 'https://vpc-test-456.com' | ||
PIPELINE_ROLE_ARN = 'arn=arn:aws:iam::123456789012:role/OSMigrations-aws-integ-us--osisPipelineRole123' | ||
AWS_REGION = 'us-west-2' | ||
SECRET_NAME = 'unit-test-secret' | ||
INDEX_INCLUSION_RULE_1 = 'index*' | ||
INDEX_INCLUSION_RULE_2 = '.*' | ||
INDEX_INCLUSION_RULE_3 = 'sam[a-z]+' | ||
|
||
|
||
class TestOSIMigration(unittest.TestCase): | ||
def test_construct_config_sigv4_source_and_sigv4_target(self): | ||
generated_config = osiMigration.construct_pipeline_config( | ||
pipeline_config_file_path='./osiPipelineTemplate.yaml', | ||
source_endpoint=SOURCE_ENDPOINT, | ||
target_endpoint=TARGET_ENDPOINT, | ||
source_auth_type='SIGV4', | ||
target_auth_type='SIGV4', | ||
pipeline_role_arn=PIPELINE_ROLE_ARN, | ||
aws_region=AWS_REGION) | ||
with open('./test/resources/sigv4SourceAndSigv4Target.yaml', "r") as expected_file: | ||
expected_file_contents = expected_file.read() | ||
self.assertEqual(generated_config, expected_file_contents) | ||
|
||
def test_construct_config_basic_auth_source_and_sigv4_target(self): | ||
generated_config = osiMigration.construct_pipeline_config( | ||
pipeline_config_file_path='./osiPipelineTemplate.yaml', | ||
source_endpoint=SOURCE_ENDPOINT, | ||
target_endpoint=TARGET_ENDPOINT, | ||
source_auth_type='BASIC_AUTH', | ||
source_auth_secret=SECRET_NAME, | ||
target_auth_type='SIGV4', | ||
pipeline_role_arn=PIPELINE_ROLE_ARN, | ||
aws_region=AWS_REGION) | ||
with open('./test/resources/basicAuthSourceAndSigv4Target.yaml', "r") as expected_file: | ||
expected_file_contents = expected_file.read() | ||
self.assertEqual(generated_config, expected_file_contents) | ||
|
||
def test_construct_config_basic_auth_source_with_single_index_inclusion_rules(self): | ||
generated_config = osiMigration.construct_pipeline_config( | ||
pipeline_config_file_path='./osiPipelineTemplate.yaml', | ||
source_endpoint=SOURCE_ENDPOINT, | ||
target_endpoint=TARGET_ENDPOINT, | ||
source_auth_type='BASIC_AUTH', | ||
source_auth_secret=SECRET_NAME, | ||
include_index_regex_list=[INDEX_INCLUSION_RULE_1], | ||
target_auth_type='SIGV4', | ||
pipeline_role_arn=PIPELINE_ROLE_ARN, | ||
aws_region=AWS_REGION) | ||
with open('./test/resources/basicAuthSourceWithSingleIndexInclusionRule.yaml', "r") as expected_file: | ||
expected_file_contents = expected_file.read() | ||
self.assertEqual(generated_config, expected_file_contents) | ||
|
||
def test_construct_config_basic_auth_source_with_multiple_index_inclusion_rules(self): | ||
generated_config = osiMigration.construct_pipeline_config( | ||
pipeline_config_file_path='./osiPipelineTemplate.yaml', | ||
source_endpoint=SOURCE_ENDPOINT, | ||
target_endpoint=TARGET_ENDPOINT, | ||
source_auth_type='BASIC_AUTH', | ||
source_auth_secret=SECRET_NAME, | ||
include_index_regex_list=[INDEX_INCLUSION_RULE_1, INDEX_INCLUSION_RULE_2, INDEX_INCLUSION_RULE_3], | ||
target_auth_type='SIGV4', | ||
pipeline_role_arn=PIPELINE_ROLE_ARN, | ||
aws_region=AWS_REGION) | ||
with open('./test/resources/basicAuthSourceWithMultipleIndexInclusionRule.yaml', "r") as expected_file: | ||
expected_file_contents = expected_file.read() | ||
self.assertEqual(generated_config, expected_file_contents) | ||
|
||
def test_construct_config_throws_error_if_secret_not_provided_for_basic_auth(self): | ||
with self.assertRaises(osiMigration.InvalidAuthParameters): | ||
osiMigration.construct_pipeline_config( | ||
pipeline_config_file_path='./osiPipelineTemplate.yaml', | ||
source_endpoint=SOURCE_ENDPOINT, | ||
target_endpoint=TARGET_ENDPOINT, | ||
source_auth_type='BASIC_AUTH', | ||
target_auth_type='SIGV4', | ||
pipeline_role_arn=PIPELINE_ROLE_ARN, | ||
aws_region=AWS_REGION) | ||
|
||
def test_construct_config_throws_error_if_pipeline_role_not_provided_for_sigv4(self): | ||
with self.assertRaises(osiMigration.InvalidAuthParameters): | ||
osiMigration.construct_pipeline_config( | ||
pipeline_config_file_path='./osiPipelineTemplate.yaml', | ||
source_endpoint=SOURCE_ENDPOINT, | ||
target_endpoint=TARGET_ENDPOINT, | ||
source_auth_type='SIGV4', | ||
target_auth_type='SIGV4', | ||
aws_region=AWS_REGION) | ||
|
||
|
||
if __name__ == '__main__': | ||
unittest.main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters