Skip to content

Unobserved Task Exception after closing ClientWebSocket #80116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mus65 opened this issue Jan 3, 2023 · 15 comments · May be fixed by #114689
Open

Unobserved Task Exception after closing ClientWebSocket #80116

mus65 opened this issue Jan 3, 2023 · 15 comments · May be fixed by #114689
Assignees
Labels
area-System.Net bug in-pr There is an active PR which will close this issue when it is merged tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Milestone

Comments

@mus65
Copy link
Contributor

mus65 commented Jan 3, 2023

Description

We randomly get the following unobserved task exception in our logs:

Unobserved Task Exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..)
 ---> System.IO.IOException: Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..
 ---> System.Net.Sockets.SocketException (995): Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.ReadBufferedAsyncCore(Memory`1 destination)
   at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

I managed to narrow this down to the ClientWebSocket close handshake and can reproduce this (see Reproduction Steps below). It happens when the closing of the connection takes too long after the close handshake finished. I think the issue is here:

await finalReadTask.AsTask().WaitAsync(TimeSpan.FromMilliseconds(WaitForCloseTimeoutMs)).ConfigureAwait(false);

When WaitAsync() throws a TimeoutException, it gets ignored, but the exception from the original finalReadTask (which may happen later) is never observed in this case.

Reproduction Steps

using System.Net.WebSockets;

namespace WebApplication2
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            TaskScheduler.UnobservedTaskException += (obj, args) =>
            {
                Console.Error.WriteLine("Unhandled exception: " + args.Exception);
            };

            var builder = WebApplication.CreateBuilder(args);
            var app = builder.Build();

            app.UseWebSockets();

            app.Use(async (HttpContext context, RequestDelegate next) =>
            {
                WebSocket webSocket = await context.WebSockets.AcceptWebSocketAsync();
                await webSocket.ReceiveAsync(new ArraySegment<byte>(new byte[4096]), CancellationToken.None);
                await webSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "normal", CancellationToken.None);
                await Task.Delay(10_000);
            });

            await app.StartAsync();

            ClientWebSocket clientWebSocket = new ClientWebSocket();
            await clientWebSocket.ConnectAsync(new Uri("ws://localhost:5065"), CancellationToken.None);
            await clientWebSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "normal", CancellationToken.None);
            await clientWebSocket.ReceiveAsync(new ArraySegment<byte>(new byte[4096]), CancellationToken.None);
            clientWebSocket.Dispose();
            await Task.Delay(5_000);
            GC.Collect(2);
            GC.WaitForPendingFinalizers();
            await Task.Delay(100_000);
        }
    }
}

Expected behavior

No unobserved task exception.

Actual behavior

Unobserved task exception:

Unobserved Task Exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..)
 ---> System.IO.IOException: Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..
 ---> System.Net.Sockets.SocketException (995): Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.ReadBufferedAsyncCore(Memory`1 destination)
   at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

Regression?

No response

Known Workarounds

No response

Configuration

.NET 6.0.11 on Windows 11 x64.

Other information

No response

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jan 3, 2023
@ghost
Copy link

ghost commented Jan 3, 2023

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

We randomly get the following unobserved task exception in our logs:

Unobserved Task Exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..)
 ---> System.IO.IOException: Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..
 ---> System.Net.Sockets.SocketException (995): Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.ReadBufferedAsyncCore(Memory`1 destination)
   at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

I managed to narrow this down to the ClientWebSocket close handshake and can reproduce this (see Reproduction Steps below). It happens when the closing of the connection takes too long after the close handshake finished. I think the issue is here:

await finalReadTask.AsTask().WaitAsync(TimeSpan.FromMilliseconds(WaitForCloseTimeoutMs)).ConfigureAwait(false);

When WaitAsync() throws a TimeoutException, it gets ignored, but the exception from the original finalReadTask (which may happen later) is never observed in this case.

Reproduction Steps

using System.Net.WebSockets;

namespace WebApplication2
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            TaskScheduler.UnobservedTaskException += (obj, args) =>
            {
                Console.Error.WriteLine("Unhandled exception: " + args.Exception);
            };

            var builder = WebApplication.CreateBuilder(args);
            var app = builder.Build();

            app.UseWebSockets();

            app.Use(async (HttpContext context, RequestDelegate next) =>
            {
                WebSocket webSocket = await context.WebSockets.AcceptWebSocketAsync();
                await webSocket.ReceiveAsync(new ArraySegment<byte>(new byte[4096]), CancellationToken.None);
                await webSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "normal", CancellationToken.None);
                await Task.Delay(10_000);
            });

            await app.StartAsync();

            ClientWebSocket clientWebSocket = new ClientWebSocket();
            await clientWebSocket.ConnectAsync(new Uri("ws://localhost:5065"), CancellationToken.None);
            await clientWebSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "normal", CancellationToken.None);
            await clientWebSocket.ReceiveAsync(new ArraySegment<byte>(new byte[4096]), CancellationToken.None);
            clientWebSocket.Dispose();
            await Task.Delay(5_000);
            GC.Collect(2);
            GC.WaitForPendingFinalizers();
            await Task.Delay(100_000);
        }
    }
}

Expected behavior

No unobserved task exception.

Actual behavior

Unobserved task exception:

Unobserved Task Exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..)
 ---> System.IO.IOException: Unable to read data from the transport connection: Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen..
 ---> System.Net.Sockets.SocketException (995): Der E/A-Vorgang wurde wegen eines Threadendes oder einer Anwendungsanforderung abgebrochen.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.ReadBufferedAsyncCore(Memory`1 destination)
   at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

Regression?

No response

Known Workarounds

No response

Configuration

.NET 6.0.11 on Windows 11 x64.

Other information

No response

Author: mus65
Assignees: -
Labels:

area-System.Net

Milestone: -

@MihaZupan
Copy link
Member

Thanks for the great repro, this does look like a bug.
Introduced by #56282 (in .NET 6.0).

This should be fairly easy to fix (that task should be awaited in the catch block after the call to Abort).
@mus65 would you be interested in contributing a PR?

@MihaZupan MihaZupan added bug and removed untriaged New issue has not been triaged by the area owner labels Jan 3, 2023
@MihaZupan
Copy link
Member

Triage: UnobservedTaskException are annoying and this one should be simple to fix, moving to 8.0 for now.

@MihaZupan MihaZupan added this to the 8.0.0 milestone Jan 3, 2023
@mus65
Copy link
Contributor Author

mus65 commented Jan 4, 2023

@MihaZupan But the await would need to wrapped in another try/catch, right? Otherwise we would throw here which currently cant' happen.

Would this change require a test case? If not, I can make a PR. I'm not sure If I could write a consistent test case for this (especially since I assume I can't use ASP .NET Core in runtime tests).

@MihaZupan
Copy link
Member

MihaZupan commented Jan 4, 2023

I think the cleanest option would be to revert to how this was written before #56282, something like

using var finalCts = new CancellationTokenSource(WaitForCloseTimeoutMs);
using (finalCts.Token.UnsafeRegister(static s => ((ManagedWebSocket)s!).Abort(), this))
{
    try
    {
        await finalReadTask;
    }
    catch { }
}

Re: tests, it's generally harder to write ones that verify that all task exceptions are observed. While having a regression test to make sure we don't reintroduce the same mistake in the future would be nice, we'd take a PR even without it.

We don't use ASP.NET in our tests, but we have our own "loopback server". You can see some examples of how that might look like here: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.WebSockets.Client/tests/SendReceiveTest.cs.

@stephentoub
Copy link
Member

stephentoub commented Jan 4, 2023

I think the cleanest option would be to revert to how this was written before #56282, something like

finalReadTask is a ValueTask. Wouldn't your suggestion result in it possibly being consume twice? Or have I misunderstood what you're suggesting?

@stephentoub
Copy link
Member

stephentoub commented Jan 4, 2023

While having a regression test to make sure we don't reintroduce the same mistake in the future would be nice, we'd take a PR even without it.

I think a PR needs a test. If it's worth fixing this to ensure we're not raising the event, then it's worth having a test for it.

@MihaZupan
Copy link
Member

finalReadTask is a ValueTask. Wouldn't your suggestion result in it possibly being consume twice? Or have I misunderstood what you're suggesting?

I meant we'd replace the current logic (WaitAsync) with setting up the CTS and await the value task once like in my example above. I don't think that could lead to awaiting twice?

@stephentoub
Copy link
Member

I meant we'd replace the current logic (WaitAsync) with setting up the CTS and await the value task once like in my example above. I don't think that could lead to awaiting twice?

Ah, replacing it would be fine from the perspective of waiting for it twice.

However, there is no guarantee that Dispose'ing of the Stream will result in the task returned from ReadAsync completing. That's why it's written the way it is.

@MihaZupan
Copy link
Member

However, there is no guarantee that Dispose'ing of the Stream will result in the task returned from ReadAsync completing. That's why it's written the way it is.

We're relying on this being the case in HttpConnection though, no?

@stephentoub
Copy link
Member

Yes, but only because at the time it was written we had no other option, e.g. there was no cancellation story for NetworkStream / Socket. I believe there's an open issue to switch HttpConnection over to using CancellationTokens threaded through all the calls instead of this approach with disposing streams, which is not thread-safe and not guaranteed to work, especially with ConnectCallback able to supply an arbitrary Stream.

@karelz karelz added the tenet-reliability Reliability/stability related issue (stress, load problems, etc.) label Jun 9, 2023
@karelz karelz modified the milestones: 8.0.0, 9.0.0 Jul 18, 2023
@wfurt
Copy link
Member

wfurt commented Jun 12, 2024

triage: not critical for 9.0, moving to future

@wfurt wfurt modified the milestones: 9.0.0, Future Jun 12, 2024
@flagbug
Copy link

flagbug commented Jun 25, 2024

We're seeing the same issue in our application and our Sentry error reporting is picking up these unobserved task exceptions. Took forever to determine that this is not a problem in our own code, but in the framework 😬

@LeadAssimilator
Copy link

Same issue with Udp/TcpClient. After closing it, every pending receive operation started with BeginReceive will emit the aforementioned unobserved task exception.

Just ran into this on maui ios where the unobserved task handler was attempting to break if a debugger was attached, which due to some other bug, just hangs 95%+ of the time - what fun that was tracking down.

This bug should have been fixed by now...having the runtime generate these kinds of errors just wastes everyone's time and needlessly increases SNR. We rely on the UnobservedTaskException to point out where we are being idiotic, but if the runtime is being an idiot too, then we really can't reliably use it as a safety check.

rampaa added a commit to rampaa/JL that referenced this issue Sep 17, 2024
…otnet/runtime#80116), so let's not forcefully terminate JL just because an UnobservedTaskException is thrown
@MihaZupan MihaZupan marked this as a duplicate of #110998 Dec 30, 2024
@kirsan31
Copy link

kirsan31 commented Mar 5, 2025

Same problem with simple Socket.Close call during Socket.BeginReceive (.NET9):

UnobservedTaskException System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)

Took some time to investigate the root cause :( ...

And documentation clearly says that all must be ok:

Close the Socket to cancel a pending BeginReceive. When the Close method is called while an asynchronous operation is in progress, the callback provided to the BeginReceive method is called. A subsequent call to the EndReceive method will throw an ObjectDisposedException (before .NET 7) or a SocketException (on .NET 7+) to indicate that the operation has been cancelled.

@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-System.Net bug in-pr There is an active PR which will close this issue when it is merged tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants