Arm request timeouts on an event loop by pavel-ptashyts · Pull Request #2313 · AsyncHttpClient/async-http-client

pavel-ptashyts · 2026-08-17T18:49:36Z

Problem

Request and read timeouts are armed on the client's HashedWheelTimer. That has two
properties that only show up on short deadlines:

A wheel quantizes. It fires on the first tick at or after the deadline, so a
deadline near or below hashedWheelTimerTickDuration is rounded up to it.
One thread carries every expiry for the whole client, and HashedWheelTimer's
default taskExecutor is ImmediateExecutor, so each expiry runs inline on the wheel
thread - including future.completeExceptionally(...) and therefore whatever the caller
chained onto the response future.

On a one-second budget the first costs 0.3% and nobody notices. On a budget of tens of
milliseconds a tick is a large fraction of it, and a burst of expiries has no headroom to
absorb before the wheel starts running late.

Measured

2000 timeouts armed as one burst on Netty 4.2.16, JDK 17, tasks doing nothing but
recording their own lag. This is the floor; real work on the firing thread only adds to it.

instrument	deadline	mean	p50	p99	max
wheel, tick 5 ms	20 ms	+2.7	+2	+5	+5
wheel, tick 1 ms	20 ms	+1.3	+1	+2	+2
`EventLoop.schedule`	20 ms	+0.0	+0	+0	+0
wheel, tick 5 ms	1000 ms	+3.0	+3	+3	+3
wheel, tick 1 ms	1000 ms	+1.9	+2	+2	+2
`EventLoop.schedule`	1000 ms	+1.6	+2	+2	+2

An event loop shows zero overshoot because it schedules by deadline and derives its own
select() timeout from the nearest one. There is no quantum to round to.

This was a throwaway probe rather than JMH - client/src/jmh/java is not currently wired
into the build, so its benchmarks do not compile. Happy to add a proper benchmark if that
is fixed first, or as part of this.

Change

AsyncHttpClientConfig#isUseEventLoopTimeouts(), off by default, arms the request and
read timeouts on an event loop instead of the timer.

Always the channel's own loop. On the pooled path the channel is already in hand, so
its loop is used and the timeout expires on the thread that would have to close it. On the
connect path there is no channel yet - deliberately, so that the timeout also bounds
address resolution and the connect - so it is armed on the timer and moved onto the loop
once the connect succeeds. No other loop is ever used; see the review round below for why
that matters.
Arming allocates nothing extra. The cancellation handle lives on the task rather than
in a wrapper, and the existing done flag stands in for the scheduler's already-expired
flag, which the two schedulers spell differently.
Shutdown race closed. isShuttingDown() can return false and schedule reject
immediately after. Netty answers a rejected timeout with a logged warning rather than an
exception, which would leave the exchange with nothing to end it, so a rejection falls
back to the timer.
The connection-pool cleaner stays on the timer either way.

Off by default because the expiry - and so whatever the caller chained onto the future -
then runs on an I/O thread, and blocking one stalls every connection it serves. The javadoc
says so and points callers at handleAsync.

Why not a wheel per event loop

That is how the Aerospike client solves the same problem: EventLoopBase owns a
HashedWheelTimer that is a Runnable the loop ticks itself. Deliberately not copied here.
A wheel arms in O(1) against O(log n) for a deadline queue, but at a few thousand timeouts
per loop that is a dozen comparisons, while the quantization it reintroduces costs
milliseconds on a 20 ms budget - the third row above is the whole point. A wheel also has to
be ticked forever, waking every loop even with nothing armed. Aerospike wrote its own because
its EventLoop abstracts over NIO, Netty and direct NIO and needed one timer; AHC is
Netty-only and gets a per-loop deadline queue for free.

Review round 1

Most of the substance of this PR changed in review, so the sections above describe the
current shape rather than what was first pushed. Two things are worth calling out here
because they were design errors, not polish:

The loop is now only ever the channel's own. It used to come from
EventLoopGroup#next(), which is almost never the loop the channel ends up on:
initAndRegister draws from the same chooser, so the two agreed about one time in N. Every
completion then cancelled an entry on a foreign loop, and until the original deadline that
entry sat in a queue whose loop it would wake for a request that had long finished. Drawing
from the chooser also shifted which loops connections land on. The pooled path has the
channel in hand; the connect path arms on the timer, as before this branch, and
NettyConnectListener moves the timeouts onto the loop once the connect succeeds, next to
the attachChannel that publishes it on the future. That listener already runs on the
channel's loop, so the move costs a same-thread schedule and no wakeup.

Arming left the TimeoutsHolder constructor. The task holds the holder and can run the
moment it is armed, and an event loop does not round a short deadline up to a tick, so the
expiry could reach a holder whose fields were not yet frozen, a future that had not been
handed the holder, and on the pooled path a future with no channel attached - which aborted
with null and left the pooled socket open. The caller now publishes the holder, attaches
the channel, and calls start() last.

Also from the review: arm re-checks cancelled after recording its handle, so an exchange
that finishes mid-arming cannot leave behind an entry nobody will cancel; cancelArmed
catches the RejectedExecutionException that Netty's off-loop cancellation path can raise on
a closing client, which had never escaped ListenableFuture#cancel before; the cancellation
handle is two typed fields rather than an Object and instanceof; requestTimeoutArmed is
gone in favour of a null test on the task; the rationale lives on
isUseEventLoopTimeouts() alone; and the new option sits in the // timeouts group
everywhere rather than splitting the two failedIpCooldown entries.

API compatibility

No revapi entries. implements Runnable and run() live on the two subclasses rather than
on TimeoutTimerTask, which leaves that class's surface unchanged: both subclasses already
declared run(Timeout) without a throws clause, so inside a subclass run() calls its own
override and has nothing to catch. No dead handler, and nothing narrowed.

The knock-on is that only the concrete classes are both a TimerTask and a Runnable, so
TimeoutsHolder#arm takes that intersection as a type parameter. Everything else is
additive: the existing TimeoutsHolder constructor is kept and delegates, and nothing is
removed.

start() is called from NettyResponseFuture#setTimeoutsHolder rather than by the sender.
TimeoutsHolder has a public constructor in an exported package, and splitting the arming
out of it would otherwise leave an outside caller free to install a holder and get an
exchange with no request timeout at all.

Tests

Four cases in EventLoopTimeoutTest, asserting where an expiry is delivered from rather than
what it does. They hand the config their own Timer and EventLoopGroup so the assertions
are against those objects and not against thread names, which a pool name containing timer
or a configured thread factory would have broken with no bug present:

the timer default, by identity against the timer's own thread;
a connecting exchange, on the loop of the channel onTcpConnectSuccess reported;
an exchange on a pooled channel, on the loop of the channel onConnectionPooled reported,
which is also what says it reused the connection rather than opening one of its own;
a read timeout under the new mode, which is armed after the request is written and so
exercises a different arming path.

The group has eight loops. With two, a timeout armed on the wrong loop is on the right one
half the time, and these assertions would have passed about half the runs against the bug they
exist to catch; at eight, reverting timeoutExecutor to next() fails the pooled case.

The connecting case, and the pooled case's first request, get a one second budget on purpose.
A deadline reached before connecting would be delivered from the timer quite correctly, there
being no channel to deliver it from, and would prove nothing either way; and the pooled case's
first request is the cold one - class loading, the connect, the server's first response - and
is meant to succeed.

One gap, called out rather than papered over: the RejectedExecutionException fallback in
arm has no test. Reaching it needs a loop that answers isShuttingDown() with false and
then rejects the schedule, and the executor comes from the channel, so there is no way in
through the config. A test double for EventExecutor would do it if that is acceptable.

Verification

mvnw clean verify - BUILD SUCCESS, 1468 tests, 0 failures, 0 errors, 21 skipped. Error
Prone, NullAway clean; revapi clean with the one scoped entry above.

Caveat on the testing gate: AGENTS.md requires the build to run on JDK 11 and no JDK 11 is
installed on this machine, so it was run on JDK 17 (also in the CI matrix). The JDK 11
leg of CI on this PR is the real gate.

Claude Code on behalf of @pavel-ptashyts

🤖 Generated with Claude Code

A hashed wheel fires on the first tick at or after a deadline, so a deadline near or below the tick duration is rounded up to it, and one timer thread carries every expiry for the whole client. Both hurt short deadlines: a tick is a large fraction of the budget, and a burst of expiries has no headroom to absorb. Measured over 2000 timeouts armed as one burst on Netty 4.2.16, a 20 ms deadline overshot by a mean of 2.7 ms and a p99 of 5 ms on a 5 ms wheel, 1.3/2 ms on a 1 ms wheel, and 0/0 ms scheduled on an event loop, which derives its select timeout from the nearest deadline and so rounds nothing. Add isUseEventLoopTimeouts(), off by default, which arms the request and read timeouts on an event loop instead. On the pooled path the channel is already in hand, so its own loop is used and the timeout expires on the thread that would have to close it. On the connect path there is no channel yet, deliberately, so that the timeout also bounds address resolution and the connect: any loop will do there, since what the wheel costs is a single thread and a rounded-up tick rather than the identity of the thread. Deliberately not a wheel per event loop, which is how the Aerospike client solves this. A wheel arms in O(1) against O(log n) for a deadline queue, but at a few thousand timeouts per loop that is a dozen comparisons, while the quantization it reintroduces costs milliseconds on a 20 ms budget; it also has to be ticked forever, waking every loop even with nothing armed. Aerospike wrote its own wheel because its EventLoop abstracts over NIO, Netty and direct NIO and needed one timer; AHC is Netty-only and gets a per-loop deadline queue for free. Arming allocates nothing beyond what the scheduler needs: the cancellation handle lives on the task, and the existing done flag stands in for the scheduler's already-expired flag, so no per-timeout wrapper is required. Left off by default because the expiry, and therefore whatever the caller chained onto the response future, then runs on an I/O thread. Blocking one stalls every connection it serves. Claude Code on behalf of Pavel Ptashyts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

hyperxpro

Round 1

Review feedback on the commit before this one, which claimed that without a channel any loop would do because what a wheel costs is a single thread and a rounded-up tick rather than the identity of the thread. That was wrong in a way the wheel had been hiding. The loop came from EventLoopGroup#next(), which is almost never the loop the channel ends up on: initAndRegister calls next() again, so for an N loop group the two agreed about one time in N. Every completion then cancelled an entry on a foreign loop, and until the original deadline that entry sat in a queue whose loop it would wake for a request that had long finished; the read timeout re-armed across loops for the same reason. Drawing from the chooser only to pick a timeout thread also advanced the counter that assigns channels to loops, so a fixed number of draws per request could settle registrations onto a subset of them. The loop is now only ever the channel's own. The pooled path has the channel in hand. The connect path arms on the timer, as it did before this branch, because the timeout has to bound address resolution and the connect itself; NettyConnectListener then moves it onto the loop once there is a channel, next to the attachChannel that publishes it on the future. That listener already runs on the channel's loop, so the move costs a same-thread schedule and no wakeup. The holder keeps the switch itself rather than making every caller consult the config, which also spares the listener a dependency on the request sender it does not otherwise need there. Arming has also left the TimeoutsHolder constructor. The task holds the holder and can run the moment it is armed, and an event loop does not round a short deadline up to the next tick, so the expiry could reach a holder whose fields were not yet frozen and a future that had not yet been handed it. On the pooled path it could also reach a future with no channel attached, abort with null, and leave the pooled socket open. The caller now publishes the holder and attaches the channel first and calls start() last. Two smaller races: arm records its handle after scheduling, so an exchange that finished in that window left an entry nobody would ever cancel, cancel() being one shot; arm now re-checks the flag afterwards. And cancelArmed did not catch what arm catches, so a late cancel on a closing client threw RejectedExecutionException out of ListenableFuture#cancel, which had never thrown before. The rest is what the review asked for and worth no argument: two typed handles instead of an Object and instanceof, so a scheduler changing its return type is a compile error rather than a cancellation that silently stops working; requestTimeoutArmed dropped for a null test on the task, which is the shape the code had before; the throws clause off run(Timeout), since the package private constructor makes the subclass it defended against impossible; the rationale in isUseEventLoopTimeouts() alone with the other three copies linking to it; and the new option in the timeouts group everywhere rather than between the two failedIpCooldown entries. Dropping that throws clause is the one thing here revapi objects to, and the only way to keep the dead catch out. It is scoped to the single method: the change is binary compatible, and the only source-level effect would be on code catching a narrower checked exception around a direct run(Timeout) call, which nothing outside the library does and no outside subclass can even reach. The tests asserted on substrings of thread names, which a pool name containing "timer" or a configured thread factory would have broken with no bug present, and only ever exercised the no-channel branch. They now hand the config their own Timer and EventLoopGroup and assert against those: the timer's own thread by identity, and for the event loop cases that the expiry arrived on the loop of the channel the handler was told about. A pooled exchange and a read timeout are covered as well. Claude Code on behalf of Pavel Ptashyts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

pavel-ptashyts · 2026-08-21T08:33:29Z

Round 1 is addressed; thanks, the two structural ones changed the shape of this for the better.

What moved:

Timeouts are only ever armed on the loop that owns the exchange's channel. next() is gone, so the group's chooser is left to initAndRegister, and the connect path arms on the timer and is moved onto the loop by NettyConnectListener once the connect succeeds.
Arming left the TimeoutsHolder constructor for a start() the caller invokes after publishing the holder, and the pooled path attaches the channel before arming rather than after.
arm re-checks cancelled after recording the handle; cancelArmed no longer throws out of cancel() during shutdown, and holds two typed handles instead of an Object.
requestTimeoutArmed and the throws on run(Timeout) are gone, the rationale lives in isUseEventLoopTimeouts() alone, and the new option sits in the // timeouts group everywhere instead of splitting the failedIpCooldown pair.
The tests assert against the timer and the loops themselves rather than thread names, and cover the pooled branch and a read timeout. The one gap left is the RejectedExecutionException fallback in arm, noted inline.

Dropping the throws on run(Timeout) needed a revapi entry, java.method.exception.checkedRemoved, scoped to that method; the justification is in the pom and the reasoning is on the inline thread. It is the only way to keep the dead catch out, and the alternative shapes trade it for a worse difference.

./mvnw clean verify is green. Not on JDK 11 as the contract asks, only 17 - no 11 on this machine.

Review round two. TimeoutsHolder has a public constructor in an exported package, and splitting the arming out of it left an outside caller free to install a holder and get an exchange with no request timeout at all. start() is now called from NettyResponseFuture#setTimeoutsHolder, so installing a holder is what arms it and neither can be done without the other. The ordering that made the split necessary still holds: the holder is reachable from the future before its task can run, and the pooled path attaches the channel before it installs. Implementing Runnable has moved from TimeoutTimerTask down into the two subclasses, which leaves the base class byte identical and drops the revapi entry the previous commit added. Both subclasses already declared run(Timeout) without a throws clause, so within a subclass run() calls its own override and has nothing to catch - the dead handler is gone without narrowing anything. Only the concrete classes are both a TimerTask and a Runnable, so arm() takes that intersection. Three narrower ones, none reachable today and all cheap to close. armedOn kept whichever handle it was given and left the other in place, so a stale timer handle would have masked a live loop handle and left its entry in the queue holding the future until a deadline nobody was waiting for; each now clears the other. cancelIfRaced sat inside the try that guards schedule(), so if cancelArmed ever stopped swallowing a rejection it would have landed in the schedule's handler and re-armed on the timer for a finished exchange; only schedule() is inside the try now. And start() arms with the configured duration rather than the remaining time, as main did: it runs within microseconds of the constructor, and reading a wall clock twice only exposes the deadline to a step between the two reads. The remaining-time arithmetic is left where subtracting elapsed time is the point, on the re-homing path. The tests ran on a group of two loops, which made the wrong loop the right one half the time: a regression to picking any loop would have passed about half the runs. Eight now, and reverting timeoutExecutor to next() fails the pooled case on both runs tried. The pooled case also kept the 200 ms budget while its first request is the cold one -- class loading, the connect, the server's first response -- and is meant to succeed, so it now takes the same second the connecting case needs. Dropped the wait for onConnectionOffer with it: finishUpdate offers to the pool before future.done(), and done() releases the latch the test was already awaiting, so it was a wait for something that had happened. Claude Code on behalf of Pavel Ptashyts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

pavel-ptashyts · 2026-08-26T17:58:31Z

Round 2 addressed. The revapi hypothesis was right, and so was the two-loop one - thanks for both, they were not obvious from where I was sitting.

implements Runnable moved down into the two subclasses, so TimeoutTimerTask is byte identical and the revapi entry is out of the pom. No dead catch either: inside a subclass run() calls its own override, which already declared no throws. arm takes <T extends TimeoutTimerTask & Runnable> as the knock-on.
start() is called from NettyResponseFuture#setTimeoutsHolder, so an outside caller cannot install a holder and get an exchange with no request timeout.
armedOn clears the other handle; cancelIfRaced moved out of the try that guards schedule(); start() arms with the configured duration rather than reading the clock a second time.
Tests: eight loops instead of two, and reverting timeoutExecutor to next() now fails the pooled case on both runs tried. The pooled case gets the cold-path budget its first request needs, and the wait for onConnectionOffer is gone - finishUpdate offers before future.done(), so it was waiting for something already done.

./mvnw clean verify green: 1468 tests, 0 failures, API checks clean with no revapi entries added. JDK 17 locally again, no 11 on this machine; the JDK 11 legs of CI are the real gate and were green on the previous push.

hyperxpro · 2026-08-26T18:58:27Z

Thanks a lot!

pavel-ptashyts closed this Aug 17, 2026

pavel-ptashyts reopened this Aug 17, 2026

pavel-ptashyts closed this Aug 17, 2026

pavel-ptashyts reopened this Aug 17, 2026

pavel-ptashyts mentioned this pull request Aug 17, 2026

Bound an exchange by one request timeout #2314

Open

hyperxpro requested changes Aug 20, 2026

View reviewed changes

pavel-ptashyts requested a review from hyperxpro August 21, 2026 08:50

hyperxpro requested changes Aug 26, 2026

View reviewed changes

pavel-ptashyts requested a review from hyperxpro August 26, 2026 18:00

hyperxpro approved these changes Aug 26, 2026

View reviewed changes

hyperxpro merged commit c1af3a4 into AsyncHttpClient:main Aug 26, 2026
13 checks passed

pavel-ptashyts deleted the feature/event-loop-request-timeouts branch August 26, 2026 19:00

Uh oh!

Conversation

pavel-ptashyts commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Measured

Change

Why not a wheel per event loop

Review round 1

API compatibility

Tests

Verification

Uh oh!

hyperxpro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pavel-ptashyts commented Aug 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pavel-ptashyts commented Aug 26, 2026

Uh oh!

Uh oh!

hyperxpro commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pavel-ptashyts commented Aug 17, 2026 •

edited

Loading