Remove dependency on System.Security.Cryptography.Native.OpenSsl in QUIC

[vcsjones]

Over at #117465, I am trying to get rid of S.S.C.Native.OpenSsl on macOS, but System.Net.Quic current uses parts of it for matching a certificate to a hostname. This makes sense since Apple's native APIs don't offer a similar API. But good news, we have an entirely managed implementation we can use on macOS.

This pull request replaces the use of the native PAL for hostname matching with our managed implementation, X509Certificate2.MatchesHostname, on macOS.

[Copilot]

Pull Request Overview

This PR removes the dependency on System.Security.Cryptography.Native.OpenSsl for hostname matching in the QUIC library on macOS by replacing native OpenSSL certificate hostname validation with the managed X509Certificate2.MatchesHostname implementation.

• Restructures project file to exclude OpenSSL cryptography interop files from macOS builds
• Replaces native OpenSSL hostname matching with managed X509Certificate2.MatchesHostname API
• Simplifies certificate validation logic by removing unsafe native code handling

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
System.Net.Quic.csproj	Reorganizes ItemGroup conditions to exclude OpenSSL cryptography interop files from macOS builds while maintaining them for Linux/FreeBSD
CertificateValidation.OSX.cs	Replaces native OpenSSL hostname matching with managed X509Certificate2.MatchesHostname implementation

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[vcsjones]

One of the tests is failing with the hostname "\u017C\u00F3\u0142\u0107 g\u0119\u015Bl\u0105 ja\u017A\u0144. \u7EA2\u70E7. \u7167\u308A\u713C\u304D". The presence of spaces causes Uri.CheckHostName to return Unknown instead of Dns. That in turn causes X509Certificate2 to reject it.

It's not immediately clear to me if this is "right". My interpretation of the relevant RFCs is that a space is not permitted in host or DNS labels, so perhaps the test should simply omit the spaces.

[vcsjones]

It should probably also be said: using libmsquic on macOS with .NET on Apple Silicon machines is very difficult. I see the tests failing on x64 macOS because libmsquic is available, but the hardened runtime on Apple Silicon makes it very difficult to enable.

[rzikm]

Does this change mean that we can get rid of

runtime/src/libraries/System.Net.Security/src/System/Net/CertificateValidationPal.OSX.cs

Lines 10 to 48 in 8300e38

	internal static partial class CertificateValidationPal
	{
	internal static SslPolicyErrors VerifyCertificateProperties(
	SafeDeleteContext securityContext,
	X509Chain chain,
	X509Certificate2? remoteCertificate,
	bool checkCertName,
	bool isServer,
	string? hostName)
	{
	SslPolicyErrors errors = SslPolicyErrors.None;
	if (remoteCertificate == null)
	{
	errors |= SslPolicyErrors.RemoteCertificateNotAvailable;
	}
	else
	{
	if (!chain.Build(remoteCertificate))
	{
	errors |= SslPolicyErrors.RemoteCertificateChainErrors;
	}
	if (!isServer && checkCertName)
	{
	SafeDeleteSslContext sslContext = (SafeDeleteSslContext)securityContext;
	if (!Interop.AppleCrypto.SslCheckHostnameMatch(sslContext.SslContext, hostName!, remoteCertificate.NotBefore, out int osStatus))
	{
	errors |= SslPolicyErrors.RemoteCertificateNameMismatch;
	if (NetEventSource.Log.IsEnabled())
	NetEventSource.Error(sslContext, $"Cert name validation for '{hostName}' failed with status '{osStatus}'");
	}
	}
	}
	return errors;
	}

And do the same thing as we do on Unix?

runtime/src/libraries/System.Net.Security/src/System/Net/CertificateValidationPal.Unix.cs

Lines 11 to 22 in e4e866a

	internal static partial class CertificateValidationPal
	{
	internal static SslPolicyErrors VerifyCertificateProperties(
	SafeDeleteContext? _ /*securityContext*/,
	X509Chain chain,
	X509Certificate2 remoteCertificate,
	bool checkCertName,
	bool isServer,
	string? hostName)
	{
	return CertificateValidation.BuildChainAndVerifyProperties(chain, remoteCertificate, checkCertName, isServer, hostName);
	}

I looked at the code recently in the TLS 1.3 support PR and it seems to me that Interop.AppleCrypto.SslCheckHostnameMatch is doing quite a lot of work (basically rebuilding the chain again?), so this change could possibly bring perf improvement.

[rzikm]

It's not immediately clear to me if this is "right". My interpretation of the relevant RFCs is that a space is not permitted in host or DNS labels, so perhaps the test should simply omit the spaces.

I am also confused by the presence of spaces in the hostname, @krwq, your name shows up on git blame, do you remember why you put spaces in the test case?

I am personally ok with removing them

[krwq]

It's not immediately clear to me if this is "right". My interpretation of the relevant RFCs is that a space is not permitted in host or DNS labels, so perhaps the test should simply omit the spaces.

I am also confused by the presence of spaces in the hostname, @krwq, your name shows up on git blame, do you remember why you put spaces in the test case?

I am personally ok with removing them

I don't remember - that was 7 years ago... but I think per this comment: dotnet/corefx#28278 (comment)

all ASCII are legal and UTF-8 used to be legal at one point.

Looking at RFC 4366 3.1:

"HostName" contains the fully qualified DNS hostname of the server,
   as understood by the client.  The hostname is represented as a byte
   string using UTF-8 encoding [[UTF8](https://datatracker.ietf.org/doc/html/rfc4366#ref-UTF8)], without a trailing dot.

   If the hostname labels contain only US-ASCII characters, then the
   client MUST ensure that labels are separated only by the byte 0x2E,
   representing the dot character U+002E (requirement 1 in Section 3.1
   of [[IDNA](https://datatracker.ietf.org/doc/html/rfc4366#ref-IDNA)] notwithstanding).  If the server needs to match the
   HostName against names that contain non-US-ASCII characters, it MUST
   perform the conversion operation described in Section 4 of [[IDNA](https://datatracker.ietf.org/doc/html/rfc4366#ref-IDNA)],
   treating the HostName as a "query string" (i.e., the AllowUnassigned
   flag MUST be set).  Note that IDNA allows labels to be separated by
   any of the Unicode characters U+002E, U+3002, U+FF0E, and U+FF61;
   therefore, servers MUST accept any of these characters as a label
   separator.  If the server only needs to match the HostName against
   names containing exclusively ASCII characters, it MUST compare ASCII
   names case-insensitively.

"non US-ASCII" characters are a bit of a moot term for things like space so let's look at the IDNA ToASCII definition...

IDNA is defined in RFC3490 4.1:

(with my comments)

   1. If the sequence contains any code points outside the ASCII range
      (0..7F) then proceed to step 2, otherwise skip to step 3.

(we skip 2 for space)

   3. If the UseSTD3ASCIIRules flag is set, then perform these checks:

     (a) Verify the absence of non-LDH ASCII code points; that is, the
         absence of 0..2C, 2E..2F, 3A..40, 5B..60, and 7B..7F.

     (b) Verify the absence of leading and trailing hyphen-minus; that
         is, the absence of U+002D at the beginning and end of the
         sequence.

(only if UseSTD3ASCIIRules is defined we consider space (20) as invalid)

There is also RFC6066 Appendix A:

   The Server Name extension now specifies only ASCII representation,
   eliminating UTF-8.

so IMO all ASCII (0x00-0x7F) are legal for decoding given it's not guaranteed that UseSTD3ASCIIRules is used for encoding. But in theory we could block them with some sort of "strict validation" setting given it's possible to encode them.

[vcsjones]

I think we are concerned more with how hostnames match against certificates, not what is permitted in the SNI extension.

RFC 1035 does not permit spaces in DNS labels. https://datatracker.ietf.org/doc/html/rfc1035#section-2.3.1

It also doesn't permit underscores, and we know OpenSSL enforces that.

Does this change mean that we can get rid of

Maybe, assuming we can get this change though :-).

Interop.AppleCrypto.SslCheckHostnameMatch is doing quite a lot of work

Yeah Ideally we could get rid of the AppleCrypto hostname check... they don't expose a straight forward API for "Is this certificate okay for this host". The only way we could get something to work was to just build a whole chain and see if the chain had a host mismatch name error.

We should keep using the native hostname checks for OpenSSL and Windows though.

[wfurt]

would it make sense to sue the managed implementation also on Linux for consistency @vcsjones ??? Or are the OpenSSL internal check unavoidable?

[vcsjones]

would it make sense to sue the managed implementation also on Linux for consistency @vcsjones ??? Or are the OpenSSL internal check unavoidable?

OpenSSL is going to do it itself in other places, so it should be consistent with itself. Apple is already using two implementations, Security.framework and OpenSSL, so we are just changing it to a... different... two implementations.

[rzikm]

LGTM, thanks!

[krwq]

As I already mentioned offline, it should be ok to break this until we see someone actually demonstrating space is needed for their scenario (which sounds very unlikely even if technically allowed). My recommendation would be to only allow same characters UseSTD3ASCIIRules talks about but I'm ok with this as is.

[vcsjones]

Quic failures are unrelated and should be addressed by #117495.