System.NullReferenceException: at Microsoft.AspNetCore.Http.DefaultHttpRequest.set_RouteValues

[ramkumars9688]

App Version: Asp.Net Core 3.1
Hosted in Azure App service.

Null reference at Router middleware is thrown intermittently. I tried to replicate but no luck.
Based on my analysis with Debug snapshot, I could see the exception reported endpoint and the request path in the httpContext is different. Not sure if this is related to this.

For e.g, Null reference exception is reported at 'api/ABC' but when I look at the snapshot of exception, it has the route as 'api/DEF'.

The exception call stack for reference:

System.NullReferenceException:
   at Microsoft.AspNetCore.Http.DefaultHttpRequest.set_RouteValues (Microsoft.AspNetCore.Http, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Builder.RouterMiddleware+<Invoke>d__4.MoveNext (Microsoft.AspNetCore.Routing, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Uts.Web.Canvas.Admin.Middleware.UserManagementMiddleware+<InvokeAsync>d__3.MoveNext (Uts.Web.Canvas.Admin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullUts.Web.Canvas.Admin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /builds/wps/uts.web.canvas.admin/Uts.Web.Canvas.Admin/Middleware/UserManagementMiddleware.csUts.Web.Canvas.Admin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 50)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions+<>c__DisplayClass5_1+<<UseMiddlewareInterface>b__1>d.MoveNext (Microsoft.AspNetCore.Http.Abstractions, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Uts.Web.Canvas.Admin.Startup+<>c+<<Configure>b__9_0>d.MoveNext (Uts.Web.Canvas.Admin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullUts.Web.Canvas.Admin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /builds/wps/uts.web.canvas.admin/Uts.Web.Canvas.Admin/Startup.csUts.Web.Canvas.Admin, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 324)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1+<ProcessRequestAsync>d__2.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=3.0.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

Any help to identify the root cause would be appreciated.

Thanks,
Ram

[Tratcher]

#25454

That's usually caused by some bad async calls in the stack, like using async void, or not awaiting a task. Can you share the code for the custom middleware in the stack trace?

[Tratcher]

Seems ok, but note you forgot to reset the status code to 200.

For e.g, Null reference exception is reported at 'api/ABC' but when I look at the snapshot of exception, it has the route as 'api/DEF'.

This also sounds like the HttpContext being used beyond the end of the request. We've checked for the easy answers, I don't know if we'll be able to get further without a repro.

[ramkumars9688]

thanks, Tratcher for your time. Appreciate your help. let me try to isolate the relevant changes in a separate repo and share it to you

[ramkumars9688]

Hi Chris, I have given access to you for my repo. It would be really helpful to get your insights on this.
Please let me know if you have any questions.

Thanks,
Ram

[Tratcher]

After a cursory reading through the code I don't see anything else that seems like an obvious cause for this. You're going to need to find a way to get more information such as a repro with a reduced app. Are the stack traces always consistent, or are there others you can share?

Some other oddities:

This code path needs to reset the status code to 200 before re-executing the request.
https://github.com/ramkumars9688/Cat/blob/b7f9bf8e94b7d6d42c420212e74e9d320e3f626e/Uts.Web.Canvas.Admin/Startup.cs#L324-L325

All of this code is being added much later in the pipeline than I'd expect. I'd normally expect things like hsts to come at the start of the pipeline.
https://github.com/ramkumars9688/Cat/blob/b7f9bf8e94b7d6d42c420212e74e9d320e3f626e/Uts.Web.Canvas.Admin/Startup.cs#L361-L370

[ramkumars9688]

Thanks for looking into this Chris. Yes, having the repo with a reduced app would make it easy. Right now we can't replicate, hence not sure which module causes this. I will analyze further and try to breakdown the app. I will get back to you with more details.

Thanks Again!

Regards,
Ram