[Console] Backfill -- Add interface and YAML specification #728

mikaylathompson · 2024-06-18T05:23:48Z

Description

This sets the groundwork for implementing backfill backend adapters for RFS by establishing the base class/interface and logic layer.

The OSI Backfill implementation already exists, but this PR does some refactoring around it. It creates--but does not implement--the RFS Backfill backends (one for docker and one for ECS).

I also laid out a starting point for the backfill section of the services.yaml (this is very much not stable and is highly subject to change as we figure out what we need) and enforce it with Cerberus schemas.

Issues Resolved

MIGRATIONS-1776 (fully completed) & MIGRATIONS-1777 (partially completed -- the remaining component is to populate the backfill info from CDK for a cloud-deployed migration console)

Testing

Unit tests added, manually tested in docker setup.

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

codecov · 2024-06-18T05:25:20Z

Codecov Report

Attention: Patch coverage is 65.62500% with 99 lines in your changes missing coverage. Please review.

Project coverage is 64.78%. Comparing base (bc78f8f) to head (e2dc574).
Report is 11 commits behind head on main.

Files	Patch %	Lines
...le/lib/console_link/console_link/logic/backfill.py	42.99%	61 Missing ⚠️
...rationConsole/lib/console_link/console_link/cli.py	47.91%	25 Missing ⚠️
.../console_link/console_link/models/backfill_base.py	83.87%	5 Missing ⚠️
...b/console_link/console_link/models/backfill_rfs.py	86.48%	5 Missing ⚠️
...b/console_link/console_link/models/backfill_osi.py	77.77%	2 Missing ⚠️
...ole/lib/console_link/console_link/logic/metrics.py	92.30%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #728      +/-   ##
============================================
+ Coverage     63.10%   64.78%   +1.67%     
- Complexity     1585     1586       +1     
============================================
  Files           226      238      +12     
  Lines          9219     9879     +660     
  Branches        771      771              
============================================
+ Hits           5818     6400     +582     
- Misses         2991     3069      +78     
  Partials        410      410

Flag	Coverage Δ
unittests	`64.02% <ø> (+0.91%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

chelma · 2024-06-18T13:33:11Z

...ution/src/main/docker/migrationConsole/lib/console_link/console_link/models/backfill_base.py

+        if not v.validate({"backfill": self.config}):
+            raise ValueError("Invalid config file for backfill", v.errors)
+
+    def create(self):


Not a blocker, and obviously this will all evolve, but it would probably be helpful to have a high level description of what each of these commands are intended to do embedded somewhere in here. The process of writing those explanations/comments will force understanding them well enough to express to others what possible implementations could look like, which will then feed back into the interface spec. Maybe you have that elsewhere already. That said - there is not much information to be gleaned from the interface spec in terms of what would be an acceptable implementation and why it would be acceptable. As a result, users of objects that implement Backfill will necessarily need to take on more responsibility because the boundary of the black box around the interface is necessarily smaller.

Oh, good point. Right now these descriptions pretty much just exist in my head. I'll start adding them as I go through.

chelma · 2024-06-18T13:34:50Z

...ution/src/main/docker/migrationConsole/lib/console_link/console_link/models/backfill_base.py

+    def get_status(self):
+        raise NotImplementedError
+
+    def scale(self, units: int):


I think I'm on board with this expression of capacity; an integer is inherently flexible here.

chelma · 2024-06-18T13:53:28Z

...lution/src/main/docker/migrationConsole/lib/console_link/console_link/models/backfill_rfs.py

+    def __init__(self, config: Dict, target_cluster: Cluster) -> None:
+        super().__init__(config)
+        self.target_cluster = target_cluster
+        self.docker_config = self.config["reindex_from_snapshot"]["docker"]


Even though we can use cerberus to enforce that the model only has a Docker or ECS section, it feels like we're better off having a factory or something to return the correct Backfill implementation based on the input model. That would hypothetically enforce cleaner abstractions because the caller requesting an RFSBackfill would not know which implementing class it was receiving.

What do you think?

I see now we're actually doing that elsewhere.

chelma · 2024-06-18T13:54:54Z

...lution/src/main/docker/migrationConsole/lib/console_link/console_link/models/backfill_rfs.py

+            "snapshot_repo": {"type": "string", "required": False},
+            "scale": {"type": "integer", "required": False, "min": 1}
+        },
+        "check_with": contains_one_of({'docker', 'ecs'}),


This is handy

chelma · 2024-06-18T13:58:34Z

...ution/src/main/docker/migrationConsole/lib/console_link/console_link/models/backfill_base.py

+}
+
+
+class Backfill:


It feels like we should be using ABCs to enforce contract compliance if we're using inheritance in this way. Is there a reason you have in mind not to?

Yeah, I think you're right. No good reason, mostly just hadn't gotten to it yet.

chelma · 2024-06-18T14:07:11Z

...icCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/tests/test_cli.py

+    with requests_mock.Mocker() as rm:
+        rm.get(f"{env.source_cluster.endpoint}/_cat/indices",
+               status_code=200,
+               text=source_cat_indices)
+        rm.get(f"{env.target_cluster.endpoint}/_cat/indices",
+               status_code=200,
+               text=target_cat_indices)
+        result = runner.invoke(cli, ['--config-file', str(VALID_SERVICES_YAML), 'clusters', 'cat-indices'],
+                               catch_exceptions=True)


Learned something new here - thanks!

chelma · 2024-06-18T14:13:29Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/logic/backfill.py

+def get_backfill(config: Dict, source_cluster: Optional[Cluster], target_cluster: Optional[Cluster]) -> Backfill:
+    if BackfillType.opensearch_ingestion.name in config:
+        if source_cluster is None:
+            raise ValueError("source_cluster must be provided for OpenSearch Ingestion backfill")
+        if target_cluster is None:
+            raise ValueError("target_cluster must be provided for OpenSearch Ingestion backfill")
+        logger.debug("Creating OpenSearch Ingestion backfill instance")
+        return OpenSearchIngestionBackfill(config=config,
+                                           source_cluster=source_cluster,
+                                           target_cluster=target_cluster)
+    elif BackfillType.reindex_from_snapshot.name in config:
+        if target_cluster is None:
+            raise ValueError("target_cluster must be provided for RFS backfill")
+
+        if 'docker' in config[BackfillType.reindex_from_snapshot.name]:
+            logger.debug("Creating Docker RFS backfill instance")
+            return DockerRFSBackfill(config=config,
+                                     target_cluster=target_cluster)
+        elif 'ecs' in config[BackfillType.reindex_from_snapshot.name]:
+            logger.debug("Creating ECS RFS backfill instance")
+            return ECSRFSBackfill(config=config,
+                                  target_cluster=target_cluster)


Now that I see this, I'm even more inclined to use ABCs to enforce the interface contract we're currently implying, but not guaranteeing, exists.

chelma · 2024-06-18T14:14:22Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/logic/backfill.py

+    if as_json:
+        return json.dumps(response)
+    return yaml.safe_dump(response)


Not a blocker, just curious - why do we need both, and when should I use either?

This is in preparation for the web API, which will likely want jsons for almost everything. So the idea is to eventually have a json version and optionally a non-json version for every call.

chelma · 2024-06-18T14:19:00Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/logic/backfill.py

+    raise UnsupportedBackfillTypeError(next(iter(config.keys())))
+
+
+def describe(backfill: Backfill, as_json=False) -> str:


I'm curious about this paradigm - why wrap the Backfill in this way? Having the additional layer here could be surprising to folks and I'm interested to hear what, if anything, would go wrong if someone came in and used the object directly?

As an alternative, why not have the return-type modularity exist in something like an OutputType enum or an Outputter object that the Backfill object contains?

Basically, why the logic layer?

The idea is two-fold:

The CLI and API are intended to be very thin clients, potentially generated automatically from a spec. This logic/middleware layer will be the layer that those clients interact with, so all they have to know is what functions are available from the library. Potentially, these files can even be used as the spec to generate the clients.

The CLI and API should never have to know exactly what underlying backend adapters they're dealing with. But in some cases (e.g. CLI params that only apply to one type) someone in between the CLI and the class itself needs to know what's happening under the hood. That's what can happen in this layer.

Someone who's building on top of the library directly is welcome to use the underlying classes, but then they're taking on responsibility for that.

Very open to feedback on a better way to do it.

For the return type specifically, I like the idea of standardizing it more--I'm not sure I quite understand what you mean by "an Outputter object that the Backfill object contains"

chelma · 2024-06-18T14:21:12Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/logic/backfill.py

+
+
+def create(backfill: Backfill, *args, **kwargs) -> str:
+    if isinstance(backfill, RFSBackfill):


Now we're doing runtime type checking, which is sometimes an indication that encapsulation is wonky. Why not have this behavior owned by the RFSBackfill object?

Yeah, i'm not crazy about this piece. I think you're right that this is a good case to have the class itself handle it.

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

chelma · 2024-06-18T15:42:49Z

We discussed in depth, and had the following takeaways:

We're not sure that we'll be able to maintain the common behavioral interface across all implementation types (e.g. OSI vs. RFS backfill) long term
However, we'll stick with the current approach for now and see how it goes
To make it clearer whether we are adhering to the core interface, we'll use ABCs to define the behavioral spec and know that if we have to do runtime type checking that we need to revisit this.

My personal suspicion is that longer term we'll end up ditching the ABCs and leaning into the divergent behavior of the different implementation types directly, but that's definitely not for certain and I'm curious to see how things evolve.

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

peternied

Looking good, handful of outstanding comments

TrafficCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/README.md

...Capture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/cli.py

...dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/environment.py

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

chelma

Have some non-blocking comments, but overall looks good. Thanks Mikayla!

chelma · 2024-06-18T18:28:02Z

...Capture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/cli.py

-        self.env = Environment(config_file)
+        try:
+            self.env = Environment(config_file)
+        except Exception as e:


Not a blocker, but - should we be more specific here? E.g. it isn't necessarily a user input issue if they don't have access to the file, for example.

chelma · 2024-06-18T21:10:45Z

...Capture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/cli.py

+def stop_backfill_cmd(ctx, pipeline_name):
+    exitcode, message = logic_backfill.stop(ctx.env.backfill, pipeline_name=pipeline_name)
+    if exitcode != 0:
+        raise click.ClickException(message)


Not a blocker, just curious - why raise ClickException rather than anything else?

It exits with a non-zero exit code and also displays the error "nicely" --not with a full traceback, etc. I think subclassing that exception to create custom ones could be a good way to go in the future to get more specific errors while keeping the convenience features.

Docs are here: https://click.palletsprojects.com/en/8.1.x/exceptions/#exception-handling

chelma · 2024-06-18T21:13:55Z

...ution/src/main/docker/migrationConsole/lib/console_link/console_link/models/backfill_base.py

+            raise ValueError("Invalid config file for backfill", v.errors)
+
+    @abstractmethod
+    def create(self, *args, **kwargs) -> CommandResult:


Not gonna block on it, but if we have a create, we should have a teardown too.

mikaylathompson added 5 commits June 17, 2024 21:52

Start adding backfill interface

140ab56

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

migration -> backfill, start adding interface

cafa0fd

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

Set yaml format, schemas, implement get_backfill

227a7d3

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

add to and clean up README

f2c438a

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

Add tests, update schemas

55e2d23

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

mikaylathompson requested review from AndreKurait, chelma, gregschohn, lewijacn, okhasawn, peternied and sumobrian as code owners June 18, 2024 05:23

Improve test coverage

29c7297

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

chelma requested changes Jun 18, 2024

View reviewed changes

Add basic Environment tests

4532045

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

Cleanup backfill vertical

4fbf010

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

mikaylathompson requested a review from chelma June 18, 2024 17:58

peternied reviewed Jun 18, 2024

View reviewed changes

Add table of contents to README

e2dc574

Signed-off-by: Mikayla Thompson <thomika@amazon.com>

mikaylathompson mentioned this pull request Jun 18, 2024

[Console] Populate services.yaml in CDK for ECS RFS backfill #734

Merged

4 tasks

chelma approved these changes Jun 18, 2024

View reviewed changes

peternied approved these changes Jun 18, 2024

View reviewed changes

peternied merged commit 4da96a1 into opensearch-project:main Jun 18, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Console] Backfill -- Add interface and YAML specification #728

[Console] Backfill -- Add interface and YAML specification #728

mikaylathompson commented Jun 18, 2024

codecov bot commented Jun 18, 2024 •

edited

Loading

chelma Jun 18, 2024

mikaylathompson Jun 18, 2024

chelma Jun 18, 2024

chelma Jun 18, 2024

chelma Jun 18, 2024

chelma Jun 18, 2024

chelma Jun 18, 2024

mikaylathompson Jun 18, 2024

chelma Jun 18, 2024

chelma Jun 18, 2024

chelma Jun 18, 2024

mikaylathompson Jun 18, 2024

chelma Jun 18, 2024

mikaylathompson Jun 18, 2024

chelma Jun 18, 2024

mikaylathompson Jun 18, 2024

chelma commented Jun 18, 2024 •

edited

Loading

peternied left a comment

chelma left a comment

chelma Jun 18, 2024

chelma Jun 18, 2024

mikaylathompson Jun 18, 2024

chelma Jun 18, 2024

		raise UnsupportedBackfillTypeError(next(iter(config.keys())))


		def describe(backfill: Backfill, as_json=False) -> str:



		def create(backfill: Backfill, args, *kwargs) -> str:
		if isinstance(backfill, RFSBackfill):

[Console] Backfill -- Add interface and YAML specification #728

[Console] Backfill -- Add interface and YAML specification #728

Conversation

mikaylathompson commented Jun 18, 2024

Description

Issues Resolved

Testing

Check List

codecov bot commented Jun 18, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chelma commented Jun 18, 2024 • edited Loading

peternied left a comment

Choose a reason for hiding this comment

chelma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 18, 2024 •

edited

Loading

chelma commented Jun 18, 2024 •

edited

Loading