Memory leak?

[roberttekaat-ireckonu]

In my solution I create and dispose a lot of V8ScriptEngine instances. However, now that load on my application is increasing, it turns out not all memory is released. I tried to get a grip on what is happening by comparing snapshots, but I dont see much there, so cannot help you with that.

[ClearScriptLib]

Hi @roberttekaat-ireckonu,

Unfortunately, we can't investigate this issue without a test case. If you can reproduce it via a small test program, we'll be happy to take a look.

Also, V8 is native code, so if you're examining memory snapshots, be sure to include native (unmanaged) memory.

Thanks!

[simontom]

We use a similar approach and have the same(?) problem.
There might be a problem with a leak somewhere in our codebase, but one thing we came across is tracking added host objects in the "ConditionalWeakTable" - that's what AI said, to be honest 😀. Is there any recommendation to what/how to use that? We add like tens of host objects using AddHostObject (and its close friend AddHostObject to global), some of them outlive the lifecycle of the Engine (some are the Scoped, some Transient-like, and the rest are Singletons).
AI said that might be some of the leaks because the table doesn't release its nodes and only Transient approach should be used instead. (Even tho' the value gets disposed, the key keeps the node stale in the table.)

So, might that be a true?
Are there any limitations / best practices how to add the host objects correctly?
Can there by such a problem (or any) even tho' we dispose the engine every time after the run finishes?

[ClearScriptLib]

Hi @simontom,

tracking added host objects in the "ConditionalWeakTable"

That really shouldn't be necessary – and it wouldn't be effective anyway.

We add like tens of host objects using AddHostObject (and its close friend AddHostObject to global), some of them outlive the lifecycle of the Engine (some are the Scoped, some Transient-like, and the rest are Singletons).

That should all be fine.

Are there any limitations / best practices how to add the host objects correctly?

In most cases, no. ClearScript is designed not to require any special treatment for scriptable .NET objects except in highly unusual situations. Under normal circumstances, such objects should be able to outlive script engines, be handed off to multiple script engines, etc.

However, ClearScript may have a bug, and we'll be happy to investigate it, but we need a test case, or at least some information to go on. For example, how are you detecting the leak? What ClearScript usage patterns trigger it? What kind of memory is leaking – managed, native, or both? What objects are leaking? A small test application that demonstrates the problem would be ideal.

Thanks!

[simontom]

Thank you for the response.
That's what I thought, too - no specialties needed for the host objects' lifecycle x engine lifecycle.

When I have some more info about that and if it's really touching the ClearScript, I'll post it here.

Currently, I have another question about memory handling. Per what I've read before, it seems, calling

engine.CollectGarbage(true);
engine.Dispose();

might help with quicker freeing of the memory. Is this idea totaly odd, or ....?

[ClearScriptLib]

Hi @simontom,

Per what I've read before, it seems, calling

engine.CollectGarbage(true);
engine.Dispose();

might help with quicker freeing of the memory. Is this idea totaly odd, or ....?

V8 garbage collection occurs at the runtime level. If your engine uses a private runtime – that is, if you created it via the V8ScriptEngine constructor – then calling CollectGarbage just before Dispose is redundant. On the other hand, if you're disposing an engine that shares its runtime with others, doing so might make sense.

Consider also that, if a V8 runtime holds references to .NET objects, disposing it could leave those objects unreachable. If they represent significant memory use, it might make sense to trigger .NET garbage collection immediately after disposing the runtime.

Good luck!

[simontom]

I'll try to keep this:

public void ForceCollection_(string context)
{
    try
    {
        var beforeTotal = GC.GetTotalMemory(forceFullCollection: false);
        var beforeGen0 = GC.CollectionCount(0);
        var beforeGen1 = GC.CollectionCount(1);
        var beforeGen2 = GC.CollectionCount(2);

        logger.LogDebug(
            "GC.Collect initiated [{Context}] - Before: {BeforeMemory}, Gen0: {Gen0}, Gen1: {Gen1}, Gen2: {Gen2}",
            context,
            FormatBytes(beforeTotal),
            beforeGen0,
            beforeGen1,
            beforeGen2
        );

        // Double-pass collection to ensure that all finalizers are run and clean up all objects
        GC.Collect(GC.MaxGeneration, GCCollectionMode.Aggressive, blocking: true, compacting: true);
        GC.WaitForPendingFinalizers();
        GC.Collect(GC.MaxGeneration, GCCollectionMode.Aggressive, blocking: true, compacting: true);

        var afterTotal = GC.GetTotalMemory(forceFullCollection: false);
        var afterGen0 = GC.CollectionCount(0);
        var afterGen1 = GC.CollectionCount(1);
        var afterGen2 = GC.CollectionCount(2);

        var freed = beforeTotal - afterTotal;
        var gen0Collected = afterGen0 - beforeGen0;
        var gen1Collected = afterGen1 - beforeGen1;
        var gen2Collected = afterGen2 - beforeGen2;

        logger.LogInformation(
            "GC.Collect completed [{Context}] - Freed: {FreedMemory}, After: {AfterMemory}, Collections: Gen0={Gen0}, Gen1={Gen1}, Gen2={Gen2}",
            context,
            FormatBytes(freed),
            FormatBytes(afterTotal),
            gen0Collected,
            gen1Collected,
            gen2Collected
        );
    }
    catch (Exception ex)
    {
        logger.LogWarning(ex, "Failed to perform GC.Collect [{Context}]", context);
    }
}

However, there is some leak inside which we try to fix.
The way we use the engine:

• Request comes
• Create new Engine
• Create the scoped service factory
• Add host objects got from the scoped factory to the engine
• We pass e.g. strings (and other types of the data) here and there between engine<->C#
• When the Javascript finishes, we call the engine.Dispose
• Do some communication with the server (send the result and such)
• Dispose the scoped service factory
  .... and repeate

Maybe, some junk stays in the unmanaged memory or something, still investigating
Or we accidentally keep the engine reference :-/ somewhere

[simontom]

Does the Engine cleans after itself even the TypedArrays?
We use

private static void CreateUint8ArrayProvider(IScriptEngine engine)
{
    dynamic uint8ArrayFactoryFunc = engine.Evaluate(
        //language=JavaScript
        """
        (function (length) {
            return new Uint8Array(length);
        }).valueOf()
        """);

    Uint8ArrayProvider.Value = new StrongBox<Func<int, ITypedArray<byte>>>(byteCount => uint8ArrayFactoryFunc(byteCount));
}```

Do we need to track the `ITypedArray` by ourselves and then dispose it, or ....?

[ClearScriptLib]

Hi @simontom,

Do we need to track the ITypedArray by ourselves and then dispose it, or ....?

Script objects don't keep their parent engines in memory. When you dispose a V8 engine – or, more accurately, a V8 runtime – its heap is destroyed, and all outstanding script object proxies on the .NET side become unbound – that is, valid but nonfunctional.

An unbound script object proxy does consume a small bit of unmanaged memory, but that should be deallocated when the proxy is collected. As with any disposable object, you can force it to release its unmanaged resources by disposing it, but that usually isn't necessary when the only such resource is a small memory block.

Unless you're keeping your unbound proxies alive and reachable, they're unlikely to be the source of your memory leak.

EDIT: One possibility is unmanaged heap fragmentation. By creating a separate V8 runtime for each request, you could be exercising the unmanaged heap very aggressively under load or over time. High-traffic server applications can mitigate this issue by reusing V8 runtimes and/or recycling worker processes.

Cheers!

[simontom]

Well, one thing solved. However, it looks like there is something staying in the unmanaged memory. But I'm not sure how to find out

[timbze]

I've also been fighting what I thought was a memory leak. Right now my theory is that it is native heap fragmentation. I'm following the advice to use jemalloc from the following article to see if that will solve my problem. Some good info in the article.

https://medium.com/@kalyanjv/three-weeks-in-the-trenches-hunting-a-4gb-native-memory-leak-that-net-couldnt-see-0713935776d0

[timbze]

Turns out my issue was not ClearScript at all. I suspected it because of native memory usage. This was my issue. config.net using Castle ProxyGenerator. In scoped services it keeps building proxies, dotnet-counters shows assemblies in memory keep counting.