Strange issues when mixing [UnmanagedCallersOnly] with [DLLImport] - "Method's type signature is not PInvoke compatible" emitted for incorrect methods

[kkukshtel]

Description

I'm working with some bindings for sokol and am trying to bind a function callback. There is a logger on the struct I'm binding that has the following signature:

public unsafe partial struct sapp_logger {
    public delegate* unmanaged[Cdecl]<sbyte*, uint, uint, sbyte*, uint, sbyte*, void*, void> func;
    public void* user_data;
}

The library provides a default logger to attach that is also in the Pinvoke code with the following signature:

[DllImport("libs/sokol", CallingConvention = CallingConvention.Cdecl, ExactSpelling = true)]
[UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
public static extern void slog_func([NativeTypeName("const char *")] sbyte* tag, [NativeTypeName("uint32_t")] uint log_level, [NativeTypeName("uint32_t")] uint log_item, [NativeTypeName("const char *")] sbyte* message, [NativeTypeName("uint32_t")] uint line_nr, [NativeTypeName("const char *")] sbyte* filename, void* user_data);

I'm binding the function as such here:

static internal void Boot() {
  unsafe
  {
      byte[] bytes = System.Text.Encoding.UTF8.GetBytes("my window");
      fixed (byte* ptr = bytes)
      {
          //init
          sapp_desc desc = default;
          desc.width = 1920;
          desc.height = 1080;
          desc.icon.sokol_default = 1;
          desc.window_title = (sbyte*)ptr;
          desc.init_cb = &Initialize;
          desc.event_cb = &Event;
          desc.frame_cb = &Frame;
          desc.cleanup_cb = &Cleanup;
          desc.logger.func = &Log.slog_func; //<----------------- here
          App.run(&desc);
      }
  }
}

Worth noting that Event/Frame/Cleanup are defined like this:

[UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
private static unsafe void Event(sapp_event* e) {}

When I run the code, the first time the native internal binding code encounters some error/warning and attempts to use the bound logger, the program fails and throws:
Unhandled exception. System.NotSupportedException: Method's type signature is not PInvoke compatible.

However, the error is thrown for the entry function into the native layer, at any point where the error may occur, even if the actual PInvoke signature is correct.

The "erroring" function call was:
Gfx.make_pipeline(&pipeline_desc);
Bound as:

[DllImport("libs/sokol", CallingConvention = CallingConvention.Cdecl, EntryPoint = "sg_make_pipeline", ExactSpelling = true)]
public static extern sg_pipeline make_pipeline([NativeTypeName("const sg_pipeline_desc *")] sg_pipeline_desc* desc);

From the C header:

sg_pipeline sg_make_pipeline(const sg_pipeline_desc* desc);

And notably, other methods that follow this same header/binding convention do not throw this same signature error:

[DllImport("libs/sokol", CallingConvention = CallingConvention.Cdecl, EntryPoint = "sg_make_image", ExactSpelling = true)]
public static extern sg_image make_image([NativeTypeName("const sg_image_desc *")] sg_image_desc* desc);

[DllImport("libs/sokol", CallingConvention = CallingConvention.Cdecl, EntryPoint = "sg_make_sampler", ExactSpelling = true)]
public static extern sg_sampler make_sampler([NativeTypeName("const sg_sampler_desc *")] sg_sampler_desc* desc);

I spent forever trying to debug the pipeline struct, but realized over time that the function/struct were totally fine. It was an issue with internal calls throwing their internal errors, that reported up to the logger.

Rewrapping the logger as:

[UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
static unsafe void Logger(sbyte* tag, uint log_level, uint log_item, sbyte* message, uint line_nr, sbyte* filename, void* user_data)  {
    delegate* unmanaged[Cdecl]<sbyte*, uint, uint, sbyte*, uint, sbyte*, void*, void> ptr = &Log.slog_func;
    ptr(tag, log_level, log_item, message, line_nr, filename, user_data);
}

Will throw the same error at the ptr call. (Thanks @mmozeiko).

It seems like it is a bad idea to try and mix UnmanagedCallersOnly and DllImport maybe? I'm not aware of any documentation as such. But if there's any request here it's for a more elucidating error message to be reported when this problem is encountered, as I went on a wild goose chase for a long time trying to debug unrelated parts of the PInvoke code because it made it sound like the signature was incorrect.

Weirdly enough, you can "fix" the issue by binding the logger like this instead:

desc.logger.func = (delegate* unmanaged[Cdecl]<sbyte*, uint, uint, sbyte*, uint, sbyte*, void*, void>)NativeLibrary.GetExport(NativeLibrary.Load("sokol"), "slog_func");

Reproduction Steps

The best way to debug this would be to use a native library that calls some function internally that can be "improperly bound` in the managed layer. I'm attaching a zip here of the whole project (it's pretty small). This is a library, so if you just build this and bind to some Program.cs that calls Engine.Init() you'll reproduce the issue.

Dinghy.Core.zip

Expected behavior

I would have expected the binding to work as is, but if not that I would have expected a better error message to tell me what was actually failing, not that the Pinvoke signature for an unrelated method was incorrect.

Actual behavior

When run, the program reports that the method signature for the entry point function in the manged layer has the wrong Pinvoke signature, which is incorrect. The signature is right.

Regression?

No response

Known Workarounds

Binding directly as:

`desc.logger.func = (delegate* unmanaged[Cdecl]<sbyte*, uint, uint, sbyte*, uint, sbyte*, void*, void>)NativeLibrary.GetExport(NativeLibrary.Load("sokol"), "slog_func");`

For the logger makes the program work instead of using extern Pinvoke code.

Configuration

Windows 64bit, .NET 7.0

Other information

Bindings are generated via https://github.com/dotnet/ClangSharp cc @tannergooding

I think the bindings are correct here, it's more that I'm unclear the best way to bind delegate* unmanaged[Cdecl]<> to an extern func tagged with [UnmanagedCallersOnly] (and if you can even do that).

[tannergooding]

func tagged with [UnmanagedCallersOnly] (and if you can even do that)

I don't believe that is supported (and ClangSharp shouldn't be generating that itself, you'd have to explicitly tell it to via --with-attribute). CC. @jkoritzinsky

DllImport is a managed entry point that ends up invoking the exported native function. If you need a function pointer for that then you should acquire it directly via NativeLibrary.GetExport as otherwise it is doing managed->unmanaged->managed->unmanaged to invoke the raw underlying export.

[kkukshtel]

@tannergooding I am adding the UnmangedCallersOnly attribute manually (hence my recent PR to ClangSharp). I am using the NativeLibrary.GetExport now, but would also still suggest that throwing:

Unhandled exception. System.NotSupportedException: Method's type signature is not PInvoke compatible.

At the call site of an unrelated function that just happens to be the entry into the native layer is a bad experience, as the calling method being thrown with the exception has the correct signature.

Also to your point, there isn't anything I've read on MSDN or elsewhere that says that doing managed->unmanaged->managed->unmanaged would result in bad behavior. Is this just common knowledge or is there some docs that say you shouldn't do that/why it wouldn't work?

[tannergooding]

It's not that doing managed->unmanaged->managed->unmanaged would cause bad behavior. It's that its doing a lot of needless work and you can just indirectly go from managed->unmanaged if you get the function pointer directly via NativeLibrary.GetExport.

The method annotated as DllImport is itself managed and basically just invokes the function pointer that's resolved by the runtime as well as doing any marshalling required and transitioning the GC. So it is managed->unmanaged

A function annotated as [UnmanagedCallersOnly] can only be invoked from unmanaged code (such as via a function pointer) and so it has a pre-stub that causes the relevant GC transition. So it is unmanaged->managed.

This also means that if you invoke such a function pointer from managed, you end up with managed->unmanaged->managed and thus you do a needless set of GC transitions.

The reason you can't mix UnmanagedCallersOnly + DllImport is that the runtime has never set that up, because its representative of the user likely doing something wrong (it's adding pointless indirections and GC transitions that provide no benefit). If you really really wanted to do that, you could define a managed wrapper method that is annotated as such and which calls the DllImport method.

[mmozeiko]

At the call site of an unrelated function that just happens to be the entry into the native layer is a bad experience, as the calling method being thrown with the exception has the correct signature.

I believe if you enable mixed mode debugging https://learn.microsoft.com/en-us/visualstudio/debugger/how-to-debug-in-mixed-mode?view=vs-2022#enable-mixed-mode-debugging-for-a-managed-calling-app then you would see that you can step into make_pipeline function just fine (so its arguments were marshalled just fine. But only when you would step into logger.fun callback, then marshalling to managed world would fail with that error.

[kkukshtel]

It's not that doing managed->unmanaged->managed->unmanaged would cause bad behavior. It's that its doing a lot of needless work and you can just indirectly go from managed->unmanaged if you get the function pointer directly via NativeLibrary.GetExport.

The method annotated as DllImport is itself managed and basically just invokes the function pointer that's resolved by the runtime as well as doing any marshalling required and transitioning the GC. So it is managed->unmanaged

A function annotated as [UnmanagedCallersOnly] can only be invoked from unmanaged code (such as via a function pointer) and so it has a pre-stub that causes the relevant GC transition. So it is unmanaged->managed.

This also means that if you invoke such a function pointer from managed, you end up with managed->unmanaged->managed and thus you do a needless set of GC transitions.

The reason you can't mix UnmanagedCallersOnly + DllImport is that the runtime has never set that up, because its representative of the user likely doing something wrong (it's adding pointless indirections and GC transitions that provide no benefit). If you really really wanted to do that, you could define a managed wrapper method that is annotated as such and which calls the DllImport method.

Just so it's not lost though, I realize that in practice my use case was incorrect, but the issue remains that I feel like the error itself that was thrown by the runtime was misleading. The function signature of make_pipeline was correct, directly contradicting the error as written, Method's type signature is not PInvoke compatible.

The function pointer should have been invoked the proper way (as mentioned above), but even then I don't think that improper function pointer use on an unrelated struct should cause an unrelated function to error in a way that says its this functions fault, no?

[mmozeiko]

Unless you are in mixed debugger, the runtime will show all errors related to doing C# callbacks at place where you go into native call. As that is what it sees on managed call stack.

The exception is not saying "make_pipeline function is wrong". It is showing callstack that leads to exception. But if you're looking only at managed callstack - you won't see where it is in native code.

[kkukshtel]

It actually looks like this line here states that PInvoke mixed with UnmanagedCallersOnly is not supported as a first-class thing, and throws the method signature error:

runtime/src/coreclr/vm/dllimport.cpp

Line 4320 in 5149896

if (pMD->HasUnmanagedCallersOnlyAttribute())

But I'm still getting the error though at make_pipeline, not at the improperly defined method. I get what you're saying @mmozeiko but that would imply that any native interop error could only ever throw "Method's type signature is not PInvoke compatible", which I don't think is the case?

[mmozeiko]

Yes, that is the place this exception happens. It is same thing what happens on ptr(...) line here:

[UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
static unsafe void Logger(sbyte* tag, uint log_level, uint log_item, sbyte* message, uint line_nr, sbyte* filename, void* user_data)  {
    delegate* unmanaged[Cdecl]<sbyte*, uint, uint, sbyte*, uint, sbyte*, void*, void> ptr = &Log.slog_func;
    ptr(tag, log_level, log_item, message, line_nr, filename, user_data);
}

As managed code is not allowed to call such functions.

But I'm still getting the error though at make_pipeline, not at the improperly defined method.

That is expected and how everything else works. You get these kind of exceptions at last managed->native transition. Not only related to UnmanagedCallers attribute.

[AaronRobinsonMSFT]

The function signature of make_pipeline was correct, directly contradicting the error as written, Method's type signature is not PInvoke compatible.

The calling convention of a method is apart of the signature.

[kkukshtel]

The function signature of make_pipeline was correct, directly contradicting the error as written, Method's type signature is not PInvoke compatible.

The calling convention of a method is apart of the signature.

Just so it's clear, this was the stack trace:

That method looks like this:

[DllImport("libs/sokol", CallingConvention = CallingConvention.Cdecl, EntryPoint = "sg_make_pipeline", ExactSpelling = true)]
public static extern sg_pipeline make_pipeline([NativeTypeName("const sg_pipeline_desc *")] sg_pipeline_desc* desc);

I was hesitant to include the actual extern at all in this report because it's just noise of the underlying "issue". The entry point, make_pipeline, has the correct signature. Nothing is wrong about it's signature. You can replicate the same error by just calling another function before or after make_pipeline that also errors in the native code. For example, doing an improper config after make_pipeline in my code, and then calling a trivial native function will do this:

Even though the function is defined as:

[DllImport("libs/sokol", CallingConvention = CallingConvention.Cdecl, EntryPoint = "sgl_draw", ExactSpelling = true)]
public static extern void draw();

draw() is obviously compatible.

The whole point of me filing this ticket is that the error message as read is incorrect. draw(), make_pipeline, etc. (it doesn't matter) all have the correct signature and are PInvoke compatible.

What's happening is that the internal code is calling out to an unrelated extern method that is messing up (and who's signature may indeed not be PInvoke compatible as it combines DLLImport and UnmanagedCallersOnly), but the error is reporting the entry function as the one with the error, and saying nothing about the function that is actually erroring.

So basically it's:

C# entry function A with correct signature -> Native Code -> Native code calls method B that is bound in managed code -> Log reports function A has non-compatible PInvoke signature

What I would expect to see is that method B is in the error message, not method A. Method A is correct, B is not.

[mmozeiko]

You're misunderstanding what debugger is showing to you. It does not say "method A" is not correct. It just shows call stack that triggers managed exception. It cannot show method B there because method B is not called yet.

If you enable mixed mode debugging - you would see exception happening in native code where it tries to figure how to call your managed code.