[SnapshotV2] Add snapshot size to status response for shallow V2 snapshots #15573

ltaragi · 2024-09-02T12:09:46Z

Description

The primaryStoreSize is calculated at the time of consolidation of shard level info in InternalClusterInfoService.java
This primaryStoreSize is further included in ClusterInfo. It also implements getPrimaryStoreSize, which can be called during the snapshot creation to persist this piece of information as the snapshot's size in bytes inside SnapshotInfo
This size in bytes value is read from SnapshotInfo and returned for shallow V2 snapshots
Use assertBusy to prevent flakiness in org.opensearch.snapshots.SnapshotStatusApisIT.testStatusAPICallForShallowCopySnapshot

Related Issues

Check List

Functionality includes testing.
~~API changes companion pull request created, if applicable.~~
~~Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-09-02T12:41:24Z

❌ Gradle check result for b2bda82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-09-02T12:52:16Z

❌ Gradle check result for 39248a5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-09-03T08:38:39Z

❕ Gradle check result for 002abda: UNSTABLE

TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-09-03T08:39:44Z

Codecov Report

Attention: Patch coverage is 52.38095% with 30 lines in your changes missing coverage. Please review.

Project coverage is 72.01%. Comparing base (036f6bc) to head (cefa0bc).
Report is 157 commits behind head on main.

Files with missing lines	Patch %	Lines
...in/java/org/opensearch/snapshots/SnapshotInfo.java	55.55%	6 Missing and 6 partials ⚠️
...admin/cluster/snapshots/status/SnapshotStatus.java	35.71%	8 Missing and 1 partial ⚠️
.../main/java/org/opensearch/cluster/ClusterInfo.java	45.45%	4 Missing and 2 partials ⚠️
...apshots/status/TransportSnapshotsStatusAction.java	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15573      +/-   ##
============================================
+ Coverage     71.92%   72.01%   +0.09%     
- Complexity    64400    64446      +46     
============================================
  Files          5281     5281              
  Lines        300995   301045      +50     
  Branches      43479    43492      +13     
============================================
+ Hits         216491   216802     +311     
+ Misses        66793    66432     -361     
- Partials      17711    17811     +100

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ltaragi · 2024-09-03T08:57:38Z

❕ Gradle check result for 002abda: UNSTABLE

TEST FAILURES:
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Flaky test #14408

devvratsaini · 2024-09-08T17:14:06Z

server/src/main/java/org/opensearch/action/admin/cluster/snapshots/status/SnapshotStatus.java

@@ -96,7 +99,12 @@ public class SnapshotStatus implements ToXContentObject, Writeable {
        includeGlobalState = in.readOptionalBoolean();
        final long startTime = in.readLong();
        final long time = in.readLong();
-        updateShardStats(startTime, time);
+        if (in.getVersion().onOrAfter(Version.CURRENT)) {


While checking this in 'main', please could you help add a TODO comment with the specific version this needs to be replaced with in the follow-up PR? Applicable for all places we have added 'Version.CURRENT'

…shots Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

github-actions · 2024-09-23T11:09:22Z

❌ Gradle check result for 8e7af89: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

github-actions · 2024-09-23T12:26:47Z

❕ Gradle check result for cefa0bc: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

gbbafna · 2024-09-25T09:30:38Z

server/src/main/java/org/opensearch/cluster/InternalClusterInfoService.java

@@ -115,6 +115,8 @@ public class InternalClusterInfoService implements ClusterInfoService, ClusterSt
    private volatile Map<String, FileCacheStats> nodeFileCacheStats;
    private volatile IndicesStatsSummary indicesStatsSummary;
    // null if this node is not currently the cluster-manager
+
+    private volatile long primaryStoreSize;


Looks like we already have final Map<String, Long> shardSizes; , which store shard level sizes. Can we just use this to retrieve shard sizes for v2 ?

opensearch-trigger-bot · 2024-10-25T15:22:20Z

This PR is stalled because it has been open for 30 days with no activity.

ltaragi force-pushed the primary-store-size branch from b2bda82 to 39248a5 Compare September 2, 2024 12:15

ltaragi force-pushed the primary-store-size branch from 39248a5 to 002abda Compare September 3, 2024 07:41

ltaragi added the skip-changelog label Sep 3, 2024

ltaragi marked this pull request as ready for review September 3, 2024 08:58

ltaragi requested review from anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, jainankitk, kotwanikunal, linuxpi, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners September 3, 2024 08:58

ltaragi requested a review from VachaShah as a code owner September 3, 2024 08:58

ltaragi self-assigned this Sep 3, 2024

opensearch-ci-bot mentioned this pull request Sep 6, 2024

[AUTOCUT] Gradle Check Flaky Test Report for ReactorNetty4StreamingIT #15825

Closed

devvratsaini suggested changes Sep 8, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Sep 11, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT #14408

Open

opensearch-ci-bot mentioned this pull request Sep 6, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

ltaragi added 3 commits September 23, 2024 15:14

Add primary store size to snapshot status result for shallow V2 snap…

4bdc5f2

…shots Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

Fix build

ececc75

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

Add snapshot_size_in_bytes field to XContent of SnapshotInfo

77ccc5d

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

ltaragi force-pushed the primary-store-size branch from 002abda to 8e7af89 Compare September 23, 2024 10:15

Add version change todos and fix potential breaking changes

cefa0bc

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>

ltaragi force-pushed the primary-store-size branch from 8e7af89 to cefa0bc Compare September 23, 2024 11:25

gbbafna reviewed Sep 25, 2024

View reviewed changes

prudhvigodithi mentioned this pull request Oct 16, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MixedClusterClientYamlTestSuiteIT prudhvigodithi/opensearch-build#82

Open

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SnapshotV2] Add snapshot size to status response for shallow V2 snapshots #15573

[SnapshotV2] Add snapshot size to status response for shallow V2 snapshots #15573

ltaragi commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024

github-actions bot commented Sep 2, 2024

github-actions bot commented Sep 3, 2024

codecov bot commented Sep 3, 2024 •

edited

Loading

ltaragi commented Sep 3, 2024 •

edited

Loading

devvratsaini Sep 8, 2024

github-actions bot commented Sep 23, 2024

github-actions bot commented Sep 23, 2024

gbbafna Sep 25, 2024

opensearch-trigger-bot bot commented Oct 25, 2024

[SnapshotV2] Add snapshot size to status response for shallow V2 snapshots #15573

Are you sure you want to change the base?

[SnapshotV2] Add snapshot size to status response for shallow V2 snapshots #15573

Conversation

ltaragi commented Sep 2, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Sep 2, 2024

github-actions bot commented Sep 2, 2024

github-actions bot commented Sep 3, 2024

codecov bot commented Sep 3, 2024 • edited Loading

Codecov Report

ltaragi commented Sep 3, 2024 • edited Loading

devvratsaini Sep 8, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 23, 2024

github-actions bot commented Sep 23, 2024

gbbafna Sep 25, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Oct 25, 2024

ltaragi commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 3, 2024 •

edited

Loading

ltaragi commented Sep 3, 2024 •

edited

Loading