JAMES-3929 Using http5 client instead of opensearch rest client #1648

Arsnael · 2023-07-18T10:14:57Z

No description provided.

server/testing/src/main/java/org/apache/james/util/docker/Images.java

Arsnael · 2023-07-19T02:12:38Z

https://ci-builds.apache.org/job/james/job/ApacheJames/job/opensearch-upgrade/1/testReport/junit/org.apache.james.backends.opensearch/OpenSearchIndexerTest/updateMessages/

Seems to be the same issue that @vttranlina encountered... Weird cause when I tested locally quickly yesterday it seemed ok. I will dig more today

...-common/opensearch/src/main/java/org/apache/james/backends/opensearch/OpenSearchIndexer.java

Arsnael · 2023-07-20T02:47:57Z

Hmmmm seems hanging forever same in some other tests in OpenSearchListeningMessageSearchIndexTest. As soon as you put OS on pause, the requests hangs on forever. Will check the client if we need to configure the timeout or something, related likely to the new Transport class

chibenwa · 2023-07-21T02:16:29Z

Exception in thread "OpenSearch-driver-3" java.lang.OutOfMemoryError: Java heap space
	at org.apache.hc.core5.util.ByteArrayBuffer.<init>(ByteArrayBuffer.java:54)
	at org.opensearch.client.transport.httpclient5.internal.HeapBufferedAsyncEntityConsumer.data(HeapBufferedAsyncEntityConsumer.java:91)
	at org.apache.hc.core5.http.nio.entity.AbstractBinDataConsumer.consume(AbstractBinDataConsumer.java:75)
	at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer.consume(AbstractAsyncResponseConsumer.java:141)
	at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:227)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:265)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:354)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:325)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
	at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:133)
	at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
	at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:85)
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
	at java.base/java.lang.Thread.run(Thread.java:829)

Likely related, and a nice problem to debug!

At your flame graphs / visualvm dumps!

chibenwa · 2023-07-21T02:17:54Z

https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-1648/4/testReport/junit/org.apache.james/WithCassandraBlobStoreTest/oneHundredMailsShouldBeWellReceived_GuiceJamesServer_/

Arsnael · 2023-07-21T02:30:26Z

Exception in thread "OpenSearch-driver-3" java.lang.OutOfMemoryError: Java heap space
	at org.apache.hc.core5.util.ByteArrayBuffer.<init>(ByteArrayBuffer.java:54)
	at org.opensearch.client.transport.httpclient5.internal.HeapBufferedAsyncEntityConsumer.data(HeapBufferedAsyncEntityConsumer.java:91)
	at org.apache.hc.core5.http.nio.entity.AbstractBinDataConsumer.consume(AbstractBinDataConsumer.java:75)
	at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer.consume(AbstractAsyncResponseConsumer.java:141)
	at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:227)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:265)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:354)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:325)
	at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
	at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
	at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:133)
	at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
	at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
	at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:85)
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
	at java.base/java.lang.Thread.run(Thread.java:829)

Likely related, and a nice problem to debug!

At your flame graphs / visualvm dumps!

I guessed with what I saw with perf tests yesterday... I had 9000+ users (10k users total) hanging at the end of the imap simulation...

I wanted to do some dumps and graphs today but well... can't access the perf test VM for now

vttranlina · 2023-07-21T04:52:02Z

hanging at the end of the imap simulation...

not related but don't forget the MAX_DURATION env variable

Arsnael · 2023-07-21T07:05:59Z

hanging at the end of the imap simulation...

not related but don't forget the MAX_DURATION env variable

I didn't and the simulation finished because of it. But when before finishing you see more than 9k users are still hanging, you know something is wrong :)

chibenwa · 2023-08-16T21:27:48Z

What is the status of this work?

Arsnael · 2023-08-17T03:05:17Z

What is the status of this work?

Couldn't find a way to solve it

chibenwa · 2023-08-17T06:14:46Z

backends-common/opensearch/pom.xml

+        <!-- Needed for opensearch-java dependency -->
+        <dependency>
+            <groupId>org.apache.httpcomponents.client5</groupId>
+            <artifactId>httpclient5</artifactId>
+            <version>5.1.4</version>
+        </dependency>
+        <!-- Needed for opensearch-java dependency -->
+        <dependency>
+            <groupId>org.apache.httpcomponents.core5</groupId>
+            <artifactId>httpcore5</artifactId>
+            <version>5.1.5</version>
+        </dependency>


Versions are not the same. Is this intendeed?

Yes. Just took what was on the java opensearch client side. If you look at the respective versions at maven central as well, you can see those two libs are not strictly coupled:

https://mvnrepository.com/artifact/org.apache.httpcomponents.client5/httpclient5

https://mvnrepository.com/artifact/org.apache.httpcomponents.core5/httpcore5

You can see there is no 5.1.5 version for httpclient5, it stops at 1.5.4 (similar with the current 5.2.x upper version actually)

httpcore5 is brought by httpclient5 cf https://mvnrepository.com/artifact/org.apache.httpcomponents.client5/httpclient5/5.1.4

IMO we do not need this dependency explicilty

Please do a stricter dependency analysis with maven: mvn dependency:tree -Dverbose in this module

(ad put the resulting output in a TXT file here)

Worth a try, forgot which one I added first back then to see the compiler still crying something was missing.

I forgot where I found that last time but I remember the scope of those deps in opensearch-java client was runtime, as you can see here:
deps.txt

Agreed that httpclient5 brings along httpcore5 though, unecessary indeed

Conflicts:

[INFO] | +- org.opensearch.client:opensearch-rest-client:jar:2.7.0:runtime [INFO] | | +- (org.apache.httpcomponents:httpclient:jar:4.5.14:runtime - version managed from 4.5.13; omitted for duplicate)

Likey an issue... The offender is restassured. Please force this dependency, no need to specify the version as we manage it in the root POM.

Idem forcethe version of httpcore to 4.4.16...

The conflict likely occurs because of restassured thus that would be why the leakwould only happen in tests.

Not sure to get it..? httpclient5 and httpcore5 are different libs from httpcore and httpclient? They even have different group ids. I don't see the correlation here

Reminder that the leak blew up in my face big time during perf tests on sandbox, so not only happening in our test suite

Or did i miss something?

I could see dependencies between them. Not as unlinked as one might think...

Arsnael · 2024-02-05T09:54:27Z

Hoping that the lib update fixes the previous encountered issues.

I didn't seem to encounter anymore the issue we had before on OpenSearchListeningMessageSearchIndexTest when putting OS on pause.

Still needs of course to be perf tested before being potentially approved/merged.

quantranhong1999

Read it

Arsnael · 2024-02-07T02:19:45Z

Green... I guess it's time to see the perf side of things :)

chibenwa · 2024-02-07T05:39:25Z

Don't forget to pick a scenario where OpenSearch is actually called...

Arsnael · 2024-02-07T06:18:04Z

Don't forget to pick a scenario where OpenSearch is actually called...

The full platform scenarii actually do call them... among other things. Thinking it's good enough for a first run at it.

Have something more specific in mind maybe to push it further on a 2nd test run?

Arsnael self-assigned this Jul 18, 2023

Arsnael mentioned this pull request Jul 18, 2023

Upgrade dependency opensearch-java 2.1.0 to newer linagora/james-project#4821

Open

chibenwa reviewed Jul 18, 2023

View reviewed changes

server/testing/src/main/java/org/apache/james/util/docker/Images.java Outdated Show resolved Hide resolved

Arsnael commented Jul 19, 2023

View reviewed changes

...-common/opensearch/src/main/java/org/apache/james/backends/opensearch/OpenSearchIndexer.java Outdated Show resolved Hide resolved

Arsnael force-pushed the opensearch-upgrade branch from 36d5d46 to 5eaf97a Compare July 20, 2023 08:09

vttranlina mentioned this pull request Jul 28, 2023

Experiment: header indexing and flat objects linagora/james-project#4797

Open

Arsnael mentioned this pull request Aug 1, 2023

JAMES-3929 Opensearch deps upgrades #1661

Merged

Arsnael force-pushed the opensearch-upgrade branch from 5eaf97a to 2a08702 Compare August 7, 2023 08:39

chibenwa reviewed Aug 17, 2023

View reviewed changes

Arsnael force-pushed the opensearch-upgrade branch from 2a08702 to 39fa24e Compare February 5, 2024 09:51

Arsnael changed the title ~~JAMES-3929 Upgrade to opensearch-java 2.6.0~~ JAMES-3929 Using http5 client instead of opensearch rest client Feb 5, 2024

Arsnael force-pushed the opensearch-upgrade branch from 39fa24e to e05d5b5 Compare February 6, 2024 07:20

quantranhong1999 reviewed Feb 6, 2024

View reviewed changes

Arsnael added 2 commits July 3, 2024 15:23

[Update] opensearch-java 2.10.4 => 2.11.1

b88ee38

[Update] Using http5 client instead of opensearch rest client

ce7c6d8

Arsnael force-pushed the opensearch-upgrade branch from e05d5b5 to ce7c6d8 Compare July 3, 2024 09:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JAMES-3929 Using http5 client instead of opensearch rest client #1648

JAMES-3929 Using http5 client instead of opensearch rest client #1648

Arsnael commented Jul 18, 2023

Arsnael commented Jul 19, 2023

Arsnael commented Jul 20, 2023

chibenwa commented Jul 21, 2023

chibenwa commented Jul 21, 2023

Arsnael commented Jul 21, 2023

vttranlina commented Jul 21, 2023

Arsnael commented Jul 21, 2023

chibenwa commented Aug 16, 2023

Arsnael commented Aug 17, 2023

chibenwa Aug 17, 2023

Arsnael Aug 17, 2023

chibenwa Aug 17, 2023

chibenwa Aug 17, 2023

Arsnael Aug 17, 2023

Arsnael Aug 17, 2023

chibenwa Aug 17, 2023 •

edited

Loading

Arsnael Aug 17, 2023

chibenwa Aug 17, 2023

Arsnael commented Feb 5, 2024

quantranhong1999 left a comment

Arsnael commented Feb 7, 2024

chibenwa commented Feb 7, 2024

Arsnael commented Feb 7, 2024

JAMES-3929 Using http5 client instead of opensearch rest client #1648

Are you sure you want to change the base?

JAMES-3929 Using http5 client instead of opensearch rest client #1648

Conversation

Arsnael commented Jul 18, 2023

Arsnael commented Jul 19, 2023

Arsnael commented Jul 20, 2023

chibenwa commented Jul 21, 2023

chibenwa commented Jul 21, 2023

Arsnael commented Jul 21, 2023

vttranlina commented Jul 21, 2023

Arsnael commented Jul 21, 2023

chibenwa commented Aug 16, 2023

Arsnael commented Aug 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chibenwa Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arsnael commented Feb 5, 2024

quantranhong1999 left a comment

Choose a reason for hiding this comment

Arsnael commented Feb 7, 2024

chibenwa commented Feb 7, 2024

Arsnael commented Feb 7, 2024

chibenwa Aug 17, 2023 •

edited

Loading