[BUG] org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex is flaky #11179

reta · 2023-11-13T18:36:39Z

Describe the bug
The test case org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex is flaky

org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex

java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
	at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1089)
	at org.opensearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2787)
	at org.opensearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2762)
	at org.opensearch.test.OpenSearchIntegTestCase.afterInternal(OpenSearchIntegTestCase.java:626)
	at org.opensearch.test.OpenSearchIntegTestCase.cleanUpCluster(OpenSearchIntegTestCase.java:2409)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more
	Suppressed: java.lang.AssertionError: All incoming requests on node [node_s2] should have finished. Expected 0 but got 237; pending tasks [[]]
		at org.opensearch.test.InternalTestCluster.lambda$assertRequestsFinished$44(InternalTestCluster.java:2796)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1077)
		... 42 more

Kas 13, 2023 12:08:38 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[opensearch[node_s2][generic][T#3],5,TGRP-RemoteCloneIndexIT]
java.lang.AssertionError:  inconsistent generation 
	at __randomizedtesting.SeedInfo.seed([12ED910E5B8FEFC]:0)
	at org.opensearch.index.translog.transfer.TranslogCheckpointTransferSnapshot$Builder.build(TranslogCheckpointTransferSnapshot.java:180)
	at org.opensearch.index.translog.RemoteFsTranslog.upload(RemoteFsTranslog.java:338)
	at org.opensearch.index.translog.RemoteFsTranslog.prepareAndUpload(RemoteFsTranslog.java:310)
	at org.opensearch.index.translog.RemoteFsTranslog.sync(RemoteFsTranslog.java:365)
	at org.opensearch.index.translog.InternalTranslogManager.syncTranslog(InternalTranslogManager.java:196)
	at org.opensearch.index.engine.InternalEngine.syncTranslog(InternalEngine.java:610)
	at org.opensearch.index.shard.IndexShard.postActivatePrimaryMode(IndexShard.java:3449)
	at org.opensearch.index.shard.IndexShard.activateWithPrimaryContext(IndexShard.java:3440)
	at org.opensearch.indices.recovery.RecoveryTarget.handoffPrimaryContext(RecoveryTarget.java:303)
	at org.opensearch.indices.recovery.PeerRecoveryTargetService$HandoffPrimaryContextRequestHandler.messageReceived(PeerRecoveryTargetService.java:418)
	at org.opensearch.indices.recovery.PeerRecoveryTargetService$HandoffPrimaryContextRequestHandler.messageReceived(PeerRecoveryTargetService.java:412)
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106)
	at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:471)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex" -Dtests.seed=12ED910E5B8FEFC

Expected behavior
The test must always pass

Plugins
Standard

Screenshots
N/A

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/29879/testReport/junit/org.opensearch.action.admin.indices.create/RemoteCloneIndexIT/testCreateCloneIndex/

The text was updated successfully, but these errors were encountered:

gbbafna · 2024-01-09T05:55:26Z

This should get fixed by #11720 . Verifying by running this in a loop.

reta · 2024-05-01T23:05:35Z

Reopening, the new cause now:

java.lang.AssertionError: test leaves transient cluster metadata behind
Expected: an empty collection
     but: <[indices.recovery.internal_remote_upload_timeout]>
	at __randomizedtesting.SeedInfo.seed([3DAF68DB71BA7C76:7247E2121E235FAD]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.opensearch.test.OpenSearchTestClusterRule.afterInternal(OpenSearchTestClusterRule.java:317)
	at org.opensearch.test.OpenSearchTestClusterRule.after(OpenSearchTestClusterRule.java:188)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:374)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

reta · 2024-06-03T21:50:30Z

The issue is not fixed nor gone; https://build.ci.opensearch.org/job/gradle-check/39665/testReport/junit/org.opensearch.action.admin.indices.create/RemoteCloneIndexIT/testCreateCloneIndex/

UncategorizedExecutionException[Failed execution]; nested: TranslogUploadFailedException[Failed to upload 2 files during transfer];
	at __randomizedtesting.SeedInfo.seed([D9AD7EB186615E6D:9645F478E9F87DB6]:0)
	at app//org.opensearch.action.support.AdapterActionFuture.unwrapEsException(AdapterActionFuture.java:102)
	at app//org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:57)
	at app//org.opensearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:73)
	at app//org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex(RemoteCloneIndexIT.java:80)
	at java.base@21.0.3/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.3/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at app//org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583)
Caused by: org.opensearch.index.translog.transfer.TranslogUploadFailedException: Failed to upload 2 files during transfer
	at app//org.opensearch.index.translog.transfer.TranslogTransferManager.transferSnapshot(TranslogTransferManager.java:199)
	at app//org.opensearch.index.translog.RemoteFsTranslog.upload(RemoteFsTranslog.java:426)
	at app//org.opensearch.index.translog.RemoteFsTranslog.prepareAndUpload(RemoteFsTranslog.java:409)
	at app//org.opensearch.index.translog.RemoteFsTranslog.ensureSynced(RemoteFsTranslog.java:341)
	at app//org.opensearch.index.translog.Translog.ensureSynced(Translog.java:837)
	at app//org.opensearch.index.translog.InternalTranslogManager.ensureTranslogSynced(InternalTranslogManager.java:184)
	at app//org.opensearch.index.engine.InternalEngine.ensureTranslogSynced(InternalEngine.java:605)
	at app//org.opensearch.index.shard.IndexShard.lambda$createTranslogSyncProcessor$44(IndexShard.java:4442)
	at app//org.opensearch.index.shard.IndexShard$6.write(IndexShard.java:4456)
	at app//org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:131)
	at app//org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:119)
	at app//org.opensearch.common.util.concurrent.BufferedAsyncIOProcessor.process(BufferedAsyncIOProcessor.java:80)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at java.base@21.0.3/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@21.0.3/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	... 1 more
	Suppressed: org.opensearch.index.translog.transfer.FileTransferException: java.lang.NullPointerException
		at app//org.opensearch.index.translog.transfer.BlobStoreTransferService.uploadBlob(BlobStoreTransferService.java:187)
		at app//org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobs$2(BlobStoreTransferService.java:105)
		at java.base@21.0.3/java.lang.Iterable.forEach(Iterable.java:75)
		at app//org.opensearch.index.translog.transfer.BlobStoreTransferService.uploadBlobs(BlobStoreTransferService.java:100)
		at app//org.opensearch.index.translog.transfer.TranslogTransferManager.transferSnapshot(TranslogTransferManager.java:163)
		... 15 more
	Caused by: java.lang.NullPointerException
		at java.base/java.util.Objects.requireNonNull(Objects.java:233)
		at org.opensearch.index.translog.transfer.BlobStoreTransferService.uploadBlob(BlobStoreTransferService.java:165)
		... 19 more
	Suppressed: org.opensearch.index.translog.transfer.FileTransferException: java.lang.NullPointerException
		at app//org.opensearch.index.translog.transfer.BlobStoreTransferService.uploadBlob(BlobStoreTransferService.java:187)
		at app//org.opensearch.index.translog.transfer.BlobStoreTransferService.lambda$uploadBlobs$2(BlobStoreTransferService.java:105)
		at java.base@21.0.3/java.lang.Iterable.forEach(Iterable.java:75)
		at app//org.opensearch.index.translog.transfer.BlobStoreTransferService.uploadBlobs(BlobStoreTransferService.java:100)
		at app//org.opensearch.index.translog.transfer.TranslogTransferManager.transferSnapshot(TranslogTransferManager.java:163)
		... 15 more
	Caused by: java.lang.NullPointerException
		at java.base/java.util.Objects.requireNonNull(Objects.java:233)
		at org.opensearch.index.translog.transfer.BlobStoreTransferService.uploadBlob(BlobStoreTransferService.java:165)
		... 19 more

gbbafna · 2024-07-30T13:02:37Z

Closing this as it will be tracked here now #14292

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Nov 13, 2023

reta mentioned this issue Nov 13, 2023

[Backport 2.x] Bump stefanzweifel/git-auto-commit-action from 4 to 5 #11174

Merged

kotwanikunal mentioned this issue Nov 28, 2023

Fix remote shards balancer and remove unused variables #11167

Merged

8 tasks

peternied removed the untriaged label Nov 30, 2023

ankitkala assigned gbbafna Dec 16, 2023

reta mentioned this issue Jan 18, 2024

[AUTOCUT] Gradle Check Failure on push to main #11936

Closed

gbbafna closed this as completed Feb 9, 2024

reta reopened this May 1, 2024

github-actions bot added the untriaged label May 1, 2024

reta mentioned this issue May 1, 2024

[Backport 2.x] Revert "Removing unused fetch sub phase processor initialization during fetch… (#12503)" #13491

Merged

gbbafna removed bug Something isn't working untriaged labels May 2, 2024

gbbafna mentioned this issue May 2, 2024

[Flaky test] Clear transient cluster setting at the end of the test #13508

Merged

8 tasks

sachinpkale closed this as completed in #13508 May 3, 2024

reta reopened this Jun 3, 2024

github-actions bot added the untriaged label Jun 3, 2024

This was referenced Jun 3, 2024

[Backport 2.x] Bump japicmp plugin version #13944

Merged

[Backport 2.x] 13776: allow adding query parameters to RequestOptions #13778

Merged

dblock removed the untriaged label Jun 6, 2024

This was referenced Jun 6, 2024

[Backport 2.x] Bump HdrHistogram to 2.2.2 and move the dependency version to version.properties #14040

Merged

[AUTOCUT] Gradle Check Failure on push to 2.x #14228

Closed

SwethaGuptha mentioned this issue Jun 14, 2024

[Backport 2.15] Fix unassigned shard allocation for batch mode #14339

Closed

gbbafna closed this as completed Jul 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex is flaky #11179

[BUG] org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex is flaky #11179

reta commented Nov 13, 2023

gbbafna commented Jan 9, 2024

reta commented May 1, 2024

reta commented Jun 3, 2024 •

edited

Loading

gbbafna commented Jul 30, 2024

[BUG] org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex is flaky #11179

[BUG] org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndex is flaky #11179

Comments

reta commented Nov 13, 2023

gbbafna commented Jan 9, 2024

reta commented May 1, 2024

reta commented Jun 3, 2024 • edited Loading

gbbafna commented Jul 30, 2024

reta commented Jun 3, 2024 •

edited

Loading