Attempt to execute managed code after the .NET runtime thread state has been destroyed

[stdcion]

Description

Issue occurs when using Fiber Local Storage (FLS) with .NET 9.0.5 and later on Windows x64 platforms.
The same code works correctly in .NET 9.0.4 and earlier versions (include .NET 8.0, .NET 7.0, .NET 6.0,
.NET Framework 4.5, etc.)

Any use of the FLS API results in a crash when the callback is invoked, with the following assertion failure:

CLR: Assert failure(PID 54444 [0x0000d4ac], Thread: 13976 [0x3698]): !"Attempt to execute managed code after the .NET runtime thread state has been destroyed."
    File: D:\a\_work\1\s\src\coreclr\vm\ceemain.cpp:1799 Image:
C:\Program Files\dotnet\dotnet.exe

A little clarification, in my project, I'm using a native library that requires calling a Detach method when the thread terminates. I implemented this using the FLS API, which previously worked without issues.

I would greatly appreciate any guidance on:

• Whether I'm implementing this correctly.
• If there's a recommended alternative approach for this in .NET 9.0.5+
• Any known workarounds for this issue.

Please don't hesitate to contact me if you require any additional information.

Reproduction Steps

I've prepared a public repository with a minimal project and steps for reproduction:
https://github.com/stdcion/win-fls-net9-bug

using System.Runtime.InteropServices;

namespace win_fls_net9_bug;

internal static class Program
{
    private static readonly ThreadExitCallback OnThreadExit = DoSomethingOnThreadExit;
    private static readonly int ThreadExitCallbackId = RegisterCallbackDelegate(OnThreadExit);

    [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
    private delegate void ThreadExitCallback(nint threadLocalValue);

    private static void Main(string[] args)
    {
        Console.WriteLine($".NET Version: {RuntimeInformation.FrameworkDescription}");

        var thread = new Thread(() =>
        {
            Console.WriteLine("Thread started");
            var handle = GCHandle.Alloc(42, GCHandleType.Pinned);
            // Register callback to be invoked when current thread exits.
            EnableThreadExitCallbackForCurrentThread(ThreadExitCallbackId, GCHandle.ToIntPtr(handle));
        }) { IsBackground = false };
        thread.Start();
        thread.Join(); // Wait for the thread to complete and callback to execute.
    }

    private static void DoSomethingOnThreadExit(nint threadLocalValue)
    {
        // The content of the method doesn't matter,
        // I can register an empty method, or a reference to a native method, the result is the same:
        // Attempt to execute managed code after the .NET runtime thread state has been destroyed.
        var handle = GCHandle.FromIntPtr(threadLocalValue);
        Console.WriteLine("Thread exit, thread local value: " + handle.Target);
        handle.Free();
    }

    private static int RegisterCallbackDelegate(ThreadExitCallback callback)
    {
        var callbackPtr = Marshal.GetFunctionPointerForDelegate(callback);
        var callbackId = WindowsImport.FlsAlloc(callbackPtr);

        if (callbackId == WindowsImport.FlsOutOfIndexes)
        {
            throw new InvalidOperationException("FlsAlloc failed: " + Marshal.GetLastWin32Error());
        }

        return callbackId;
    }

    private static void EnableThreadExitCallbackForCurrentThread(int callbackId, nint threadLocalValue)
    {
        var result = WindowsImport.FlsSetValue(callbackId, threadLocalValue);

        if (!result)
        {
            throw new InvalidOperationException("FlsSetValue failed: " + Marshal.GetLastWin32Error());
        }
    }

    /// <summary>
    /// P/Invoke declarations for Windows FLS (Fiber Local Storage) APIs.
    /// FLS provides thread-local storage with destructor callbacks.
    /// </summary>
    private static class WindowsImport
    {
        public const string DllName = "kernel32.dll";

        public const int FlsOutOfIndexes = -1;

        [DllImport(DllName, SetLastError = true)]
        public static extern int FlsAlloc(nint destructorCallback);

        [DllImport(DllName, SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool FlsSetValue(int key, nint threadLocalValue);

    }
}

How to run:

dotnet build --configuration Release
cd win-fls-net9-bug\bin\Release\net9.0\
dotnet --fx-version 9.0.4 .\win-fls-net9-bug.dll # works correctly
dotnet --fx-version 9.0.5 .\win-fls-net9-bug.dll # fails

Expected behavior

Assert failure does not occur:

> dotnet --fx-version 9.0.5 .\win-fls-net9-bug.dll
.NET Version: .NET 9.0.5
Thread started
Thread exit, thread local value: 42

Actual behavior

Assert failure:

> dotnet --fx-version 9.0.5 .\win-fls-net9-bug.dll
.NET Version: .NET 9.0.5
Thread started
CLR: Assert failure(PID 54444 [0x0000d4ac], Thread: 13976 [0x3698]): !"Attempt to execute managed code after the .NET runtime thread state has been destroyed."
    File: D:\a\_work\1\s\src\coreclr\vm\ceemain.cpp:1799 Image:
C:\Program Files\dotnet\dotnet.exe

Regression?

• Works correctly in .NET 9.0.4 and earlier versions (include .NET 8.0, .NET 7.0, .NET 6.0, .NET Framework 4.5, etc.)
• Does not work since .NET 9.0.5

Known Workarounds

I don't know, I would really appreciate it if you could suggest some workaround.

Configuration

• .NET Version: 9.0.5 and newer
• OS: Windows 11 Pro 24H2, Build: 26100.4349
• Arch: x64
• Release or Debug configuration does not matter

Other information

It's most likely related to this PR: #112809
with this assertion:

runtime/src/coreclr/vm/ceemain.cpp

Lines 1701 to 1717 in 496af72

	static void OsAttachThread(void* thread)
	{
	_ASSERTE(g_flsIndex != FLS_OUT_OF_INDEXES);
	if (t_flsState == FLS_STATE_INVOKED)
	{
	_ASSERTE_ALL_BUILDS(!"Attempt to execute managed code after the .NET runtime thread state has been destroyed.");
	}
	t_flsState = FLS_STATE_ARMED;
	// Associate the current fiber with the current thread. This makes the current fiber the thread's "home"
	// fiber. This fiber is the only fiber allowed to execute managed code on this thread. When this fiber
	// is destroyed, we consider the thread to be destroyed.
	_ASSERTE(thread != NULL);
	FlsSetValue(g_flsIndex, thread);
	}

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[mangod9]

we have recently fixed an issue in 9.0.8: #116872. Is it possible for you to try a private build, since it's not released yet?

[jkotas]

In general, it is not safe to execute managed code inside TLS or FLS callbacks. Executing managed code inside these callbacks always lead to hard to diagnose memory leaks or crashes.

We have added an explicit check for this to make these memory leaks and crashes easier to diagnose.

Potential workarounds:

• Make sure that your FLS callback gets registered before COM and .NET register their callbacks. It is very fragile.
• Use unmanaged code to register FLS callback, and forward these callbacks as work items to a worker thread. This only works if the work can be executed asynchronously.

[VSadov]

Also it may be possible to use threadstatic finalizable objects.

[stdcion]

Hello Team!

First of all, thank you for responding and for your advice!

I'd like to share some additional details.
In our .NET project, we integrate a native library that was built using GraalVM Native Image. Essentially, it's a Java project compiled into a dynamic library (dll, so, etc.), somewhat similar to .NET Native AOT.

Using GraalVM Native Image requires the use of a specific API, basic terms:

• Isolate - a JVM instance;
• IsolateThread - a thread attached to the current JVM instance.

Here are the main limitations:

• Every functions in the native library require IsolateThread as their first parameter.
• To obtain an IsolateThread, you must attach the current thread to the JVM.
• To prevent resource leaks, detach the thread from the JVM when it's no longer needed.
• You can only detach a thread from the same thread you are detaching, otherwise it is undefined behavior.

And a little pseudocode sample:

// For the purposes of this discussion, 
// it doesn't matter how or when Isolate was created, it could just be a singleton.
Isolate isolate = CreateGraalIsolate();

var thread = new Thread(() =>
{
    // If the thread has already been attached, 
    // this method immediately returns the attached IsolateThread.
    IsolateThread isolateThread = GetOrAttachIsolateThread(isolate);
    CallSomeJavaMethod(isolateThread, arg1, arg2);  
    DetachIsolateThread(isolateThread); // This method cannot be called from some other thread.
});
thread.Start();

I realize this is a .NET repository, not GraalVM, I'm just providing this context in case someone finds it useful.

I considered various approaches and initially didn't want to use FLS or similar solutions. However, it ultimately proved to be the most convenient option, аor end users of my library, there are no restrictions on which threads can call library methods or how they should be called and they also don't need to worry about potential resource leaks related to this. I mean no disclosure of attach/detach threads in the public API, all just implementation details.

[stdcion]

Also it may be possible to use threadstatic finalizable objects.

Unfortunately it seems not, unless I'm mistaken, finalizers are called in some separate finalizer threads, and I can only detach threads from the same thread.

[stdcion]

In general, it is not safe to execute managed code inside TLS or FLS callbacks. Executing managed code inside these callbacks always lead to hard to diagnose memory leaks or crashes.
We have added an explicit check for this to make these memory leaks and crashes easier to diagnose.

Yes, I understand this could cause problems, although I don't really need to call the managed code, it would be enough for me to pass a pointer to an unmanaged function, to detach the thread.

Perhaps we could add a property to disable this check? It could remain enabled by default.

Use unmanaged code to register FLS callback, and forward these callbacks as work items to a worker thread. This only works if the work can be executed asynchronously.

Yes, I got the idea. I'll see what can be done, thanks!

[stdcion]

we have recently fixed an issue in 9.0.8: #116872. Is it possible for you to try a private build, since it's not released yet?

Sure! I haven't actually tried private builds before, but I'm think I'll figure it out, thank you!
Do I need to build the runtime from master for this?