Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFS] RfsWorker runs through the Snapshot Phase #629

Closed
wants to merge 6 commits into from

Conversation

chelma
Copy link
Member

@chelma chelma commented May 2, 2024

Description

  • Added a new entrypoint, RunRfsWorker.java
  • Added a new class RfsWorker to run all the steps in a reindex-from-snapshot operation
  • Added a new class OpenSearchCmsClient to handle using the target cluster to coordinate work between the RFS Workers
  • Added a state machine for the individual steps of the Snapshot phase
  • Added unit tests for the Snapshot phase's steps

Issues Resolved

Testing

  • Added unit tests, which pass
  • Ran manually using gradle against the local docker-compose setup
chelma@3c22fba4e266 RFS % gradle runRfsWorker --args='--snapshot-name reindex-from-snapshot --s3-repo-uri s3://chelma-iad-rfs-local-testing --s3-region us-east-1 --source-host http://localhost:19200 --target-host http://localhost:29200'

> Task :run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
14:59:56.127 INFO  Running RfsWorker
14:59:56.129 INFO  Checking if work remains in the Snapshot Phase...
14:59:56.952 INFO  Snapshot not yet completed, entering Snapshot Phase...
14:59:56.952 INFO  Snapshot CMS Entry not found, attempting to create it...
14:59:57.248 INFO  Snapshot CMS Entry created
14:59:57.248 INFO  Attempting to initiate the snapshot...
14:59:58.756 INFO  Snapshot repo registration successful
14:59:59.044 INFO  Snapshot reindex-from-snapshot creation initiated
14:59:59.044 INFO  Snapshot in progress...
14:59:59.324 INFO  Snapshot not finished yet; sleeping for 5 seconds...
15:00:04.397 INFO  Snapshot not finished yet; sleeping for 5 seconds...
15:00:09.477 INFO  Snapshot not finished yet; sleeping for 5 seconds...
15:00:14.557 INFO  Snapshot not finished yet; sleeping for 5 seconds...
15:00:19.687 INFO  Snapshot completed, exiting Snapshot Phase...
15:00:19.687 INFO  Snapshot Phase complete

Check List

  • New functionality includes testing
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

chelma added 5 commits May 2, 2024 14:03
Signed-off-by: Chris Helma <chelma+github@amazon.com>
Signed-off-by: Chris Helma <chelma+github@amazon.com>
Signed-off-by: Chris Helma <chelma+github@amazon.com>
Signed-off-by: Chris Helma <chelma+github@amazon.com>
Signed-off-by: Chris Helma <chelma+github@amazon.com>
public void run() throws Exception {
logger.info("Running RfsWorker");

while (true) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This outer while loop is no longer necessary; a holdover from a previous iteration that I forgot to remove.

@chelma chelma removed the request for review from kartg May 3, 2024 13:01
Signed-off-by: Chris Helma <chelma+github@amazon.com>
@chelma chelma requested a review from peternied as a code owner May 3, 2024 16:56
@chelma chelma closed this May 10, 2024
@chelma chelma deleted the MIGRATIONS-1691 branch May 10, 2024 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant