latency between onMessage and onReady execution resulting in client timeout

[chkrishna411]

grpc server version used: 1.58.0
we are using spring boot starter net.devh with 2.15

We have seen timeouts from the client that the RPC was not processed within 70ms deadline.

To check the actual issue, we have increased the timeout and tested again. After implementing all the callbacks, we have observed a latency of close to 75ms between onReady() and onMessage().

Client and Server are running in the same subnet, and we are using Istio for routing the traffic b/w client and server. At the time of receiving the RPC, there was not much load as well to figure out whether it's because of thread unavailability of workers. Please check the attached image for logs.

This timeout issue is intermittent, it's not occurring every time. If we retry the request we are able to successfully process the rpc within 4-5ms.

[kannanjgithub]

The ServerCall.Listener onReady() callback only indicates that the response stream is now ready to receive messages from the server. It does not tell anything about the ability to receive incoming messages. You should instead log timestamps on the sender side such as from ClientCall.sendMessage and compare it with the timestamp logged in ServerCall.Listener::on message.

[kannanjgithub]

What is your C++ gRPC version? What does the API call look like on the client - is it async, blocking or callbacks? I think the issue you are facing may be on the gRPC C++ side.