Version
Foudn in hyper 1.5.1 and later
Platform
25.5.0 (macOS), though it is also applying on our linux bots
Summary
At OneSignal we have a number of tonic gRPC services we maintain. After a recent update of dependencies on one of them, we noticed that it "randomly" would hang during shutdown for an hour or more.
After a week of chasing the wild goose, we have isolated it to a single PR introduced in hyper 1.5.1:
#3729
In these cases a tonic server:
Server::builder()
.add_service(GreeterServer::new(MyGreeter::default()))
.serve_with_incoming_shutdown(incoming, async move {
shutdown_grpc_token.cancelled().await
})
.await
.unwrap();
would never return and block for an hour or more after the shutdown_grpc_token CancellationToken was canceled.
We have a primary example non_upgrade_connections_can_block_shutdown and a secondary one we found along the way client_not_responding_example in this self contained example:
https://github.com/chamons/hyper_hung_client_example
With a patch.crates-io line, we can resolve this branch with the PR reverted and both of these example begin acting as expected.
Expected Behavior
It is not expected that passive tonic gRPC connections to a server that are unused would block shutdown. Before 1.5.1 they did not.
Actual Behavior
We block until the upstream instance restarts or otherwise drops all of its connections.
Additional Context
No response
Version
Foudn in hyper 1.5.1 and later
Platform
25.5.0 (macOS), though it is also applying on our linux bots
Summary
At OneSignal we have a number of tonic gRPC services we maintain. After a recent update of dependencies on one of them, we noticed that it "randomly" would hang during shutdown for an hour or more.
After a week of chasing the wild goose, we have isolated it to a single PR introduced in hyper 1.5.1:
#3729
In these cases a tonic server:
would never return and block for an hour or more after the shutdown_grpc_token CancellationToken was canceled.
We have a primary example
non_upgrade_connections_can_block_shutdownand a secondary one we found along the wayclient_not_responding_examplein this self contained example:https://github.com/chamons/hyper_hung_client_example
With a
patch.crates-ioline, we can resolve this branch with the PR reverted and both of these example begin acting as expected.Expected Behavior
It is not expected that passive tonic gRPC connections to a server that are unused would block shutdown. Before 1.5.1 they did not.
Actual Behavior
We block until the upstream instance restarts or otherwise drops all of its connections.
Additional Context
No response