ProxyResponse is null in AddResponseTransform

[arnonax-tr]

Describe the bug

I'm not 100% sure it's a bug, but either it is, or something is unclear to me.

Occasionally* I'm seeing that ResponseTransformContext.ProxyResponse is null inside a delegate I pass to TransformBuilderContext.AddResponseTransform, and I don't understand how that's possible. I see that the summary of ResponseTransformContext.ProxyResponse says it "can be null if the destination did not respond", but I see from the logs of the proxied service that it did send a response (with Status code 401- Unauthorized, though I'm not sure it's related). Even if for some reason that response message was lost, then I would expect to have a way to get the error of the root cause (e.g. timeout, DNS issue, network disconnected, etc.). but I don't find any property that return that information. Note that I don't think that the problem is timeout because from the logs I see that I got the callback immediately after the proxied service responded.

*I saw it only twice out of 632 responses from the same endpoint (over a week period), and out of 9635 responses from all (3) endpoints. All of the responses from that endpoint return status 401. In most cases ResponseTransformContext.ProxyResponse is not null, but in those two cases out of 632 it is.

To Reproduce

I cannot reproduce it deterministically (and the chances to hit are very low as I mentioned above), but I can outline the structure of my code in regard to YARP. (It may be possible to reproduce it deterministically using load testing, but I don't have the time and resources for it now)

// In Bootstrapper:
private void ConfigureReverseProxy(IServiceCollection services)
{
        services.AddReverseProxy().LoadFromMemory(new[]
        {
            new RouteConfig
            {
                RouteId = "route1",
                ClusterId = "cluster1",
                Match = new RouteMatch
                {
                    Path = "{**catch-all}"
                }
            }
        },
        new[]
        {
            new ClusterConfig
            {
                ClusterId = "cluster1",
                Destinations = new Dictionary<string, DestinationConfig>
                {
                    ["destination1"] = new() { Address = ConfigurationProvider.StsUrl }
                }
            }
        })
    .AddTransforms<StsProxyTransforms>();
}

internal class MyProxyTransforms : ITransformProvider
{
    void ITransformProvider.ValidateRoute(TransformRouteValidationContext context)
    {
    }

    void ITransformProvider.ValidateCluster(TransformClusterValidationContext context)
    {
    }
    
    public void Apply(TransformBuilderContext context)
    {
        context.AddRequestTransform(async requestContext =>
        {
            // Here we examine the request and do some stuff, but we don't modify the request
            // ...
        });

        context.AddResponseTransform(async responseContext =>
        {
            // THE FOLLOWING LINE OCCASIONALLY THROWS NullReferenceException BECAUSE responseContext.ProxyResponse is null...
            if (!responseContext.ProxyResponse.IsSuccessStatusCode)
            // ...
        }
    }
}

Further technical details

• Package: Yarp.ReverseProxy (2.1.0)
• The platform where the error occurs: .Net 6 on Linux

[MihaZupan]

You can access the error via httpContext.Features.Get<IForwarderErrorFeature>().
It'll be set by the time you see a null response in a response transform.

[arnonax-tr]

Thanks @MihaZupan , I'll try that. I think it's worth mentioning in the docs of ResponseTransformContext.ProxyResponse

[arnonax-tr]

I tried it and now I see (in another endpoint BTW) the following information:

Error: RequestCanceled
Exception: System.Threading.Tasks.TaskCanceledException
Message: A task was canceled.
Stack trace:
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)

Does it give you any hint about the root cause? To me it still doesn't mean anything useful...

[MihaZupan]

reverse-proxy/src/ReverseProxy/Forwarder/ForwarderError.cs

Lines 26 to 29 in b1a1d8b

	/// <summary>
	/// Canceled when trying to connect, send the request headers, or receive the response headers.
	/// </summary>
	RequestCanceled,

RequestCanceled generally indicates that the client (e.g. browser) disconnected/canceled the request before YARP was able to get a response from the backend server.
This may be as innocent as the user navigating away to a different web page.

Updating the docs on ProxyResponse to hint at using IForwarderErrorFeature makes sense.

[arnonax-tr]

Thanks!