JsonSerializerOptions memory leak?

[shawndewet]

My ASP.Net Core 3.1 application is leaking memory.
When I take a memory dump and analyze it in DotMemory, I see the following:

If I drill into the JsonSerializerOptions link, I see that there are 3 instances of JsonSerializerOptions in memory, one of which is a clear culprit:

If I drill into that instance, I see the following shortest retention path (this is one of 128 retention paths):

Am I correct in interpreting this as follows:
JsonSerializerOptions is being held in memory by the _htmlSafeJsonSerializerOptions field of SystemTExtJsonHelper,
which is being held in memory by entry number 256 of the Dictionary<ServiceKey, Object>,
which is being held in memory by ServiceProviderEngineScope, part of the DI internals of ASP.Net Core 3.1.

The key retention paths show the following:

My app is configured to use System.Text.Json (and not Newtonsoft.Json), with the following:

services.AddControllersWithViews(options =>
{
  options.Filters.Add(new AuthorizeFilter()); //authorize everything!
})
.AddJsonOptions(opt =>
{
  opt.JsonSerializerOptions.PropertyNamingPolicy = null;
})

Please can someone help me make sense of this? Why is JsonSerializerOptions consuming so much memory in my application?

[Tratcher]

@pranavkm

[pranavkm]

My ASP.Net Core 3.1 application is leaking memory.

Do you see your applications memory grow unbounded? JsonSerializerOptions caches metadata about the types it serializes. It's usually bounded since the number of types in the application is bounded.

That said, seeing it use 800 MB is surprising. Are you able to inspect what instances are being held by JsonSerializerOptions instance?

/cc @steveharter \ @layomia

[shawndewet]

Yes if I expand the JsonSerializerOptions I can see what it is referencing in this image:

It contains a _classes field which contains a _tables field which contains a _buckets field.
This _buckets field contains over 1500 entries, each of which is over a MB in size.
The above screenshot expands two of those entries to give you an idea of what they are storing.

Does this help?

For more info about the application...it starts off at around 500MB of data usage....and yes, it grows unbounded over a period of about 6 to 8 hours, at which point it consumes all of the available 5.6GB of RAM on the server (the other 2.4GB of the total 8GB is used by other applications). So I have configured the IIS App Pool to recycle when memory usage exceeds 4.5GB.

[pranavkm]

My GitHub alias should be good enough.

[shawndewet]

Invite sent. However, I was considering your earlier comment again: "JsonSerializerOptions caches metadata about the types it serializes".
And then your hint about it serializing multiple types with over 300 properties.
The only type in the project that comes to mind is the "Debtor" domain object.
However, I was trying to understand why it is not just serializing the metadata for the one "Debtor" type....and considering why it would be seeing this as multiple types...
And I think it is because this specific type is returned to the front-end as a dynamic type. The system has functionality in which the tenant can define multiple custom Debtor properties, and then when the Debtor is retrieved, the application builds up a dynamic type which contains the standard domain properties, but also all the custom properties that the tenant has defined. Perhaps the JsonSerializerOptions caching is seeing each dynamic instance as a different "type" and therefore caches the metadata for it.
Does this make sense? I can point you to the specific method in the repo where this dynamic object is built up if you want to see the detail?

[pranavkm]

I can point you to the specific method in the repo where this dynamic object is built up if you want to see the detail?

Yes please, that would help narrow things down 👍

[shawndewet]

It's in the ToDebtorData method of the TenantSpecificController of the .MVC project.
Line 653 is where the standard domain object (returned from the database) collection gets converted to a .DataSourceResult, part of the Telerik Kendo UI MVC functionality.
That datasourceresult's .Data property then contains an array of objects.
The code after that line deals with iterating through each object in the array (converting it to dynamic), and then tagging onto the dynamic object the tenant-specific custom fields as additional properties.
Finally the method then returns that array of dynamic objects higher up, the call-chain, to be returned to the data grid as a Json result.

[steveharter]

To determine if STJ needs to address this scenario, I assume the "dynamic" object being returned is a new Type because it is actually a new CLR Type through IL Emit or other means. If so, how would STJ distinguish between these "dynamic" objects and a standard type.

Also the terminology is somewhat confusing here since the C# "dynamic" feature is normally implemented by a single Type (that implements IDynamicMetaObjectProvider or derives from DynamicObject) but that can represent any underlying shape.

So in other words, C# dynamic support should be fine but approaches that actually generate a new Type for each instance will have issues which cannot be fixed by STJ unless it can somehow detect those and not cache.

[shawndewet]

Perhaps a solution is to have the JsonSerializerOptions NOT cache metadata for dynamic types?

[shawndewet]

Thanks for the above pointers.
It turns out that in my building of the custom fields into the object being returned, it was actually returning an object and not a dynamic. This was causing the JsonSerializerOptions to cache each object result individually. I have now changed the code to ensure it returns a dynamic, and it seems this is (correctly) NOT being cached.
The unbounded memory creep seems to have dissipated.

[pranavkm]

Glad that you were able to resolve this.

@layomia FYI in case we want to do something like create an LRU cache for the serializer settings.