[API Proposal]: DefaultInitializeAttribute for out parameters in native marshallers #98049
Description
Background and motivation
If you have an existing P/Invoke that has an out parameter, that parameter is still zero initialized unless you have the calling method declared as SkipLocalsInit. Translating a P/Invoke like this to a DllImport results in a behavior change, as the generated method has SkipLocalsInit, and doesn't initialize the local passed by pointer into the backing P/Invoke. So if your API previously was depending on the out value being zeroed upon entry, this is problematic.
One of the bigger cases this is important, is if you're outting a custom struct with a custom marshaller. Many native methods will do error checking first, and not fill out variables on error. However, the custom marshaller does not know that the function errored, so it unconditionally will attempt to unmarshal the native value. Most unmarshallers could handle a zeroed struct, but would blow up with a garbage struct (Structs containing strings or arrays are the big offenders here).
To solve this, add an attribute that can be applied to out variables that will default initialize the value for that out passed into native code.
API Proposal
namespace System.Runtime.InteropServices.Marshalling;
[AttributeUsage(AttributeTargets.Parameter)]
public sealed class DefaultInitializeAttribute : Attribute
{
}API Usage
[LibraryImport("mylib", EntryPoint = "Function")]
[UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]
internal static partial void DoSomething([DefaultInitialize] out int status);Alternative Designs
Instead of DefaultInitializeAttribute, an alternative such as AlwaysMarshalAttribute could be used, where even for out variables the managed is marshalled into unmanaged. However, this breaks a fundamental contract of out variables, as the C# side would be reading from an out variable before setting it, rather than just zero initializing a variable passed down to native code.
Risks
No response