Adding a test script for snapshot restore and ability to take snapshots

[okhasawn]

Description

This PR adds a script that will create an index and a document on a source cluster, directly; not through the proxy, then creates a snapshot on the source cluster, moves the snapshot files to the target cluster, restores that snapshot on the target cluster.

After all that, a request to the target cluster is sent that attempts to update the document that already exists on the target cluster (was included in snapshot), but the update is made if and only if the timestamp in the incoming request is newer than the one that already existed.

Here's an example run of the script:

test % ./snapRestoreTest.sh
Creating an example index
{"acknowledged":true,"shards_acknowledged":true,"index":"test"}

Indexing an example document
{"_index":"test","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

Creating snapshot repo on source cluster
{"acknowledged":true}

Creating snapshot on source cluster
{"snapshot":{"snapshot":"snapshot_1","uuid":"zuDm-5PBRpyjj7ologFW3g","version_id":7100299,"version":"7.10.2","indices":["searchguard",".signals_watches_state",".searchguard_config_history","test",".searchguard_authtokens",".signals_settings",".signals_watches_trigger_state",".signals_accounts",".signals_watches"],"data_streams":[],"include_global_state":true,"state":"SUCCESS","start_time":"2023-11-15T18:03:48.687Z","start_time_in_millis":1700071428687,"end_time":"2023-11-15T18:03:48.687Z","end_time_in_millis":1700071428687,"duration_in_millis":0,"failures":[],"shards":{"total":9,"failed":0,"successful":9}}}
Copying snapshot files from source cluster to target cluster

                                              Successfully copied 119kB to /Users/okhasawn/Workspaces/opensearch-migrations/test/.
                           Successfully copied 119kB to 5dc9a809d9c5:/usr/share/opensearch/snapshots


Creating snapshot repo on target cluster
{"acknowledged":true}

Restoring from snapshot
{"accepted":true}

Checking how document looks like BEFORE the conflict resolution script is ran.
{"_index":"test","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":
{
 "field": "old_value",
 "timestamp": "2017-11-30T10:00:00"
}
}

Running conflict resolution script with new data
{"_index":"test","_id":"1","_version":2,"result":"updated","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":1,"_primary_term":2}

Checking how document looks like AFTER it was updated with script.
{"_index":"test","_id":"1","_version":2,"_seq_no":1,"_primary_term":2,"found":true,"_source":{"field":"new_value","timestamp":"2017-11-30T10:07:00"}}

Testing with a document that doesn't have a newer timestamp
{"_index":"test","_id":"1","_version":3,"result":"updated","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":2,"_primary_term":2}

Checking if document got updated after the script was ran.
{"_index":"test","_id":"1","_version":3,"_seq_no":2,"_primary_term":2,"found":true,"_source":{"field":"new_value","timestamp":"2017-11-30T10:07:00"}}%  

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b41128e) 77.04% compared to head (8dbee3f) 73.42%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #383      +/-   ##
============================================
- Coverage     77.04%   73.42%   -3.63%     
  Complexity     1109     1109              
============================================
  Files           141      116      -25     
  Lines          5769     4614    -1155     
  Branches        536      414     -122     
============================================
- Hits           4445     3388    -1057     
+ Misses         1047      964      -83     
+ Partials        277      262      -15     

Flag	Coverage Δ
unittests	73.42% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sumobrian]

Can you update this branch and post a screenshot of the results in the description?

[okhasawn]

This was closed as it acted as a POC and worked as expected.