Why are Unhandled Exception Handlers such a mess?

[fstugren]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Hi Asp.net team,

I've been a big proponent of using unhandled exception handlers in our asp.net web services. However, the more we looked at it in detail, the less the strategy works. This applies both to the "classic" asp.net error handlers enabled via UseExceptionHandler() and the modern one based on the IExceptionHandler interface.

We use custom error handlers for several things:

1. To return an error status code and error payload to the caller
2. To close some request-specific objects, i.e. issue some custom error metrics and logs

We've been aware that if an exception happens after the response has started (i.e. while writing to the output stream) the custom exception handlers are not invoked. However, recently we discovered a new confusing case - in a percentage of the requests (but not all), when the caller of the service drops the connection (i.e. HttpContext.RequestAborted is signaled) the exception handler is also not invoked.

These cases make it very hard to write a consistent error handling strategy based on the built-in error handling mechanisms. We can write error handling code that takes into consideration whether the response has already started. However, these error cases leave a significant number of failed requests outside of the default error handling, and they must be handled somehow.

I decided to remove the error handlers and wrap the top-level middleware in a custom middleware that catches and handles all exceptions from the inner middlewares and controllers, whether the response has started or not, whether the caller dropped the connection or not, and processes them the same way as the former custom error handler did. This works just fine, but it leads to the obvious question - what's the "right" thing to do?

Strangely, the asp.net runtime's unhandled exception handler is invoked every time an exception makes it through, regardless of request status, it's just the custom handlers that are skipped.

What does the Asp.Net team recommend? I need an exception handler that is guaranteed to be executed regardless of when the exception happens. I can continue to use my custom top-level middleware, but the very existence of the error handling documented mechanisms gives a misleading picture to developers. This issue has caused us to undercount errors, and it's unclear for how long. If the RequestAborted case had caused the error handler to be skipped 100%, we would have identified the issue right away, but because only some requests end up in this corner case we did not identify it until much later.

Thanks
F.

Expected Behavior

No response

Steps To Reproduce

1. Service A accepts calls from a variety of clients
2. Service A calls service B asynchronously (using standard HttpClient APIs)
3. The call to service B has a guarded timeout that involves the request cancellation token HttpContext.RequestAborted
4. Caller drops connection for whatever reason, at any time
5. If request to B is in progress at the time, it is correctly aborted and a TaskCanceledException is rethrown
6. Service A expects this exception to be caught by its unhandled exception handler where final logging/metrics are issued
7. Exception handler is not invoked. We validated this in various ways, even placed an "entered exception handler" log line as the first code in the error handler.

Exceptions (if any)

No response

.NET Version

.Net 8.0

Anything else?

No response

[MihuBot]

Possible related and/or duplicate issues:

• Consider running user-provided exception handler even when response has started #13170
• Allow ExceptionHandlerOptions.ExceptionHandler to be called even if the response has started #9839

[fstugren]

Does the asp.net team have any feedback or recommendation on this?
CC @MihaZupan

[davidfowl]

Where are you seeing the connection aborted issues? Do you have a callstack? Is it happening outside of your code?

I need an exception handler that is guaranteed to be executed regardless of when the exception happens. I can continue to use my custom top-level middleware, but the very existence of the error handling documented mechanisms gives a misleading picture to developers. This issue has caused us to undercount errors, and it's unclear for how long. If the RequestAborted case had caused the error handler to be skipped 100%, we would have identified the issue right away, but because only some requests end up in this corner case we did not identify it until much later.

What are you doing in your error handler? If you are collecting metrics, you could consider using the build in meter instead of recouting.

That said, the exception handler will catch all exceptions after where it is defined. It sounds like you want to inject the exception handler as early as possible in the pipeline to avoid missing "typical" exceptions. Still, I'd love to understand the stack where request aborted fires, that'd help us gauage if it is even feasible to catch the error (e.g. if it happens lower in the stack before your code runs, we don;t fire the callback).

[fstugren]

Hi @davidfowl thanks for following up.

I have replaced the error handler with a simple top-level middleware that catches all exceptions.

Before this change I often did not have any call stack. The CancellationToken HttpContext.RequestAborted would signal whenever the connection was be dropped (for whatever reason), and any async operation that was in process, which used HttpContext.RequestAborted as a cancellation token would throw a TaskCanceledException - this was expected. However, because of strange timing issue, the unhandled exception handler would not be invoked in many of these cases, and we missed logging errors and incrementing metrics. We noticed this when we discovered incomplete request logs that did not complete with either success or failure.

Here's an example call stack, caught now. During an outbound call that the service makes using HttpClient.SendAsync, the inbound connection is dropped. The HttpClien.SendAsync call uses an aggregated CancellationToken which signals when the inbound connection is dropped.

 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.<EnsureReadAheadTaskHasStarted>g__ReadAheadWithZeroByteReadAsync|40_0()
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Azure.ProjectBabylon.GatewayService.ProxyMiddleware.<>c__DisplayClass11_0.<<SendRequest>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.ProjectBabylon.PlatformShared.OperationManager.RunAsyncInternalWithAsyncCallbackError[T](Func`1 action, Int32 retryIntervalMilliseconds, Int32 maxAttempts, TelemetryCallbackSuccess`1 telemetryCallbackSuccess, TelemetryCallbackErrorAsync telemetryCallbackError, ExceptionHandler exceptionHandler, CancellationToken cancellationToken)
   at Microsoft.Azure.ProjectBabylon.PlatformShared.OperationManager.RunAsyncInternalWithAsyncCallbackError[T](Func`1 action, Int32 retryIntervalMilliseconds, Int32 maxAttempts, TelemetryCallbackSuccess`1 telemetryCallbackSuccess, TelemetryCallbackErrorAsync telemetryCallbackError, ExceptionHandler exceptionHandler, CancellationToken cancellationToken)
   at Microsoft.Azure.ProjectBabylon.GatewayService.ProxyMiddleware.SendRequest(HttpContext context, RewriteRuleBind bind, Uri targetUri)
   at Microsoft.Azure.ProjectBabylon.GatewayService.ProxyMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.AccountAuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.ExposureControlMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.PrivateEndpointMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.RateLimitingMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.AccountAuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.AccountResolutionMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.DelayedTokenValidationMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.GatewayForwardingMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.TenantResolutionMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.GLSTargetResolutionMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.RequestMatchingMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.TrustedCallerMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.ServiceTagEnforcerMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.GatewayMiddleware.Invoke(HttpContext context)
   at Microsoft.Azure.ProjectBabylon.GatewayService.ErrorHandlingMiddleware.Invoke(HttpContext context, Stopwatch timeTracker)
   --- End of inner exception stack trace ---