-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Fix: Validate SSL certificate private key access at server startup #7847
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: Validate SSL certificate private key access at server startup #7847
Conversation
**Problem**: Akka.Remote server starts successfully even when the application lacks permissions to access the SSL certificate's private key. The server appears healthy but fails when clients attempt to connect, making issues hard to diagnose. **Root Cause**: Certificate loading in DotNettyTransportSettings only validates that the certificate EXISTS in the Windows certificate store, not whether the application can ACCESS the private key. Private key access is checked separately by Windows ACL, which can fail even when Certificate.HasPrivateKey returns true. **Solution**: 1. Add ValidateCertificate() method to SslSettings class that: - Checks Certificate.HasPrivateKey - Actually tests private key access with GetRSAPrivateKey() (not just presence) - Throws ConfigurationException with clear error message on failure 2. Call validation in Listen() method before server socket binds: - Ensures fail-fast behavior at startup - Prevents server from running in broken state - Provides clear error message for administrators 3. Add comprehensive tests: - Server should fail at startup with inaccessible private key - Server should start successfully with valid certificate - Server should start successfully without SSL **Impact**: - Existing misconfigured deployments will now fail at startup (correct behavior) - Clear error messages guide administrators to fix permissions - No breaking changes for correctly configured systems - Related to Freshdesk akkadotnet#538 (BNSF Railway) Fixes akkadotnet#538
**Changes**: 1. Renamed first test to `Server_should_fail_at_startup_with_certificate_without_private_key` - Now validates that server FAILS AT STARTUP with bad certificate - Tests fail-fast behavior instead of runtime TLS handshake failure 2. Removed redundant `Server_side_tls_handshake_failure_should_shutdown_server` test - This test validated the OLD (incorrect) behavior where server starts successfully - Now impossible with fail-fast validation in place - Scenario already covered by the updated first test 3. Kept `Client_side_tls_handshake_failure_should_shutdown_client` unchanged - Still valid - tests client-side validation failure - Not affected by server startup validation **Result**: Tests now validate correct fail-fast behavior at server startup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Detailed my changes
// This ensures fail-fast behavior if private key is inaccessible | ||
if (Settings.EnableSsl) | ||
{ | ||
Settings.Ssl.ValidateCertificate(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Validate the private key during server binding, which should trigger shutdown of the ActorSystem
if it fails
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This spec is either somewhat redundant or unnecessary given the fail fast implementation - so it's been simplified.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this needs to be tightened.
src/core/Akka.Remote/Transport/DotNetty/DotNettyTransportSettings.cs
Outdated
Show resolved
Hide resolved
Addresses review feedback from @Arkatufus: **Changes**: 1. Check both RSA and ECDSA private keys - SslStream supports both RSA and ECDSA certificates - GetRSAPrivateKey() returns null for ECDSA certs (and vice versa) - Validation now checks both key types to match TLS handler behavior 2. Use `using` statements for proper disposal - Prevents resource leaks if exception is thrown - Both rsaKey and ecdsaKey are properly disposed - Exception-safe resource management **TLS Handler Relationship**: The TLS handler uses `TlsHandler.Server(Settings.Ssl.Certificate)` which internally extracts either RSA or ECDSA private keys via SslStream. Our validation now matches this behavior by checking both key types. **Behavior**: - RSA certificate: GetRSAPrivateKey() succeeds, GetECDsaPrivateKey() returns null ✅ - ECDSA certificate: GetECDsaPrivateKey() succeeds, GetRSAPrivateKey() returns null ✅ - Neither accessible: Both return null, validation fails with clear error ✅ - Permission denied: CryptographicException caught, clear error message ✅
@Arkatufus Good catch on the ECDSA keys! I've updated the validation to check both RSA and ECDSA private keys. Changes Made1. Added ECDSA Key ValidationNow checking both key types that SslStream supports: using (var rsaKey = Certificate.GetRSAPrivateKey())
using (var ecdsaKey = Certificate.GetECDsaPrivateKey())
{
if (rsaKey == null && ecdsaKey == null)
throw new ConfigurationException(...);
} Behavior:
2. Improved Disposal PatternChanged from manual Relationship to TLS HandlerThe TLS handler uses Note on Future Key TypesPer .NET documentation, Tests still passing ✅ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we can call this "best effort" for a fail fast detection.
This is going into the |
…kkadotnet#7847) * Fix: Validate SSL certificate private key access at server startup **Problem**: Akka.Remote server starts successfully even when the application lacks permissions to access the SSL certificate's private key. The server appears healthy but fails when clients attempt to connect, making issues hard to diagnose. **Root Cause**: Certificate loading in DotNettyTransportSettings only validates that the certificate EXISTS in the Windows certificate store, not whether the application can ACCESS the private key. Private key access is checked separately by Windows ACL, which can fail even when Certificate.HasPrivateKey returns true. **Solution**: 1. Add ValidateCertificate() method to SslSettings class that: - Checks Certificate.HasPrivateKey - Actually tests private key access with GetRSAPrivateKey() (not just presence) - Throws ConfigurationException with clear error message on failure 2. Call validation in Listen() method before server socket binds: - Ensures fail-fast behavior at startup - Prevents server from running in broken state - Provides clear error message for administrators 3. Add comprehensive tests: - Server should fail at startup with inaccessible private key - Server should start successfully with valid certificate - Server should start successfully without SSL **Impact**: - Existing misconfigured deployments will now fail at startup (correct behavior) - Clear error messages guide administrators to fix permissions - No breaking changes for correctly configured systems - Related to Freshdesk akkadotnet#538 (BNSF Railway) Fixes akkadotnet#538 * Update DotNettyTlsHandshakeFailureSpec to validate fail-fast behavior **Changes**: 1. Renamed first test to `Server_should_fail_at_startup_with_certificate_without_private_key` - Now validates that server FAILS AT STARTUP with bad certificate - Tests fail-fast behavior instead of runtime TLS handshake failure 2. Removed redundant `Server_side_tls_handshake_failure_should_shutdown_server` test - This test validated the OLD (incorrect) behavior where server starts successfully - Now impossible with fail-fast validation in place - Scenario already covered by the updated first test 3. Kept `Client_side_tls_handshake_failure_should_shutdown_client` unchanged - Still valid - tests client-side validation failure - Not affected by server startup validation **Result**: Tests now validate correct fail-fast behavior at server startup * Add ECDSA private key validation and improve disposal pattern Addresses review feedback from @Arkatufus: **Changes**: 1. Check both RSA and ECDSA private keys - SslStream supports both RSA and ECDSA certificates - GetRSAPrivateKey() returns null for ECDSA certs (and vice versa) - Validation now checks both key types to match TLS handler behavior 2. Use `using` statements for proper disposal - Prevents resource leaks if exception is thrown - Both rsaKey and ecdsaKey are properly disposed - Exception-safe resource management **TLS Handler Relationship**: The TLS handler uses `TlsHandler.Server(Settings.Ssl.Certificate)` which internally extracts either RSA or ECDSA private keys via SslStream. Our validation now matches this behavior by checking both key types. **Behavior**: - RSA certificate: GetRSAPrivateKey() succeeds, GetECDsaPrivateKey() returns null ✅ - ECDSA certificate: GetECDsaPrivateKey() succeeds, GetRSAPrivateKey() returns null ✅ - Neither accessible: Both return null, validation fails with clear error ✅ - Permission denied: CryptographicException caught, clear error message ✅
…7847) (#7848) * Fix: Validate SSL certificate private key access at server startup **Problem**: Akka.Remote server starts successfully even when the application lacks permissions to access the SSL certificate's private key. The server appears healthy but fails when clients attempt to connect, making issues hard to diagnose. **Root Cause**: Certificate loading in DotNettyTransportSettings only validates that the certificate EXISTS in the Windows certificate store, not whether the application can ACCESS the private key. Private key access is checked separately by Windows ACL, which can fail even when Certificate.HasPrivateKey returns true. **Solution**: 1. Add ValidateCertificate() method to SslSettings class that: - Checks Certificate.HasPrivateKey - Actually tests private key access with GetRSAPrivateKey() (not just presence) - Throws ConfigurationException with clear error message on failure 2. Call validation in Listen() method before server socket binds: - Ensures fail-fast behavior at startup - Prevents server from running in broken state - Provides clear error message for administrators 3. Add comprehensive tests: - Server should fail at startup with inaccessible private key - Server should start successfully with valid certificate - Server should start successfully without SSL **Impact**: - Existing misconfigured deployments will now fail at startup (correct behavior) - Clear error messages guide administrators to fix permissions - No breaking changes for correctly configured systems - Related to Freshdesk #538 (BNSF Railway) Fixes #538 * Update DotNettyTlsHandshakeFailureSpec to validate fail-fast behavior **Changes**: 1. Renamed first test to `Server_should_fail_at_startup_with_certificate_without_private_key` - Now validates that server FAILS AT STARTUP with bad certificate - Tests fail-fast behavior instead of runtime TLS handshake failure 2. Removed redundant `Server_side_tls_handshake_failure_should_shutdown_server` test - This test validated the OLD (incorrect) behavior where server starts successfully - Now impossible with fail-fast validation in place - Scenario already covered by the updated first test 3. Kept `Client_side_tls_handshake_failure_should_shutdown_client` unchanged - Still valid - tests client-side validation failure - Not affected by server startup validation **Result**: Tests now validate correct fail-fast behavior at server startup * Add ECDSA private key validation and improve disposal pattern Addresses review feedback from @Arkatufus: **Changes**: 1. Check both RSA and ECDSA private keys - SslStream supports both RSA and ECDSA certificates - GetRSAPrivateKey() returns null for ECDSA certs (and vice versa) - Validation now checks both key types to match TLS handler behavior 2. Use `using` statements for proper disposal - Prevents resource leaks if exception is thrown - Both rsaKey and ecdsaKey are properly disposed - Exception-safe resource management **TLS Handler Relationship**: The TLS handler uses `TlsHandler.Server(Settings.Ssl.Certificate)` which internally extracts either RSA or ECDSA private keys via SslStream. Our validation now matches this behavior by checking both key types. **Behavior**: - RSA certificate: GetRSAPrivateKey() succeeds, GetECDsaPrivateKey() returns null ✅ - ECDSA certificate: GetECDsaPrivateKey() succeeds, GetRSAPrivateKey() returns null ✅ - Neither accessible: Both return null, validation fails with clear error ✅ - Permission denied: CryptographicException caught, clear error message ✅
Problem
Akka.Remote server starts successfully even when the application lacks permissions to access the SSL certificate's private key. The server appears healthy but fails when clients attempt to connect, causing:
Customer Impact
A production customer reported this issue where:
Root Cause
Certificate loading in
DotNettyTransportSettings
only validates that the certificate EXISTS in the Windows certificate store, not whether the application can ACCESS the private key.Key insight: Windows separates certificate storage from private key storage with individual ACL permissions.
Certificate.HasPrivateKey
may returntrue
even when the application lacks ACL permissions to use the private key.Solution
1. Add ValidateCertificate() method to SslSettings
Certificate.HasPrivateKey
GetRSAPrivateKey()
(not just presence)ConfigurationException
with clear error message on failure2. Call validation in Listen() before server bind
3. Comprehensive test coverage
Changes
Files Modified
Akka.Remote/Transport/DotNetty/DotNettyTransportSettings.cs
ValidateCertificate()
method toSslSettings
classAkka.Remote/Transport/DotNetty/DotNettyTransport.cs
Listen()
before server socket bindsAkka.Remote.Tests/Transport/DotNettyCertificateValidationSpec.cs
Akka.Remote.Tests/Transport/DotNettyTlsHandshakeFailureSpec.cs
Impact
Breaking Change (Expected)
Existing misconfigured deployments will now fail at startup instead of silently starting with broken TLS. This is correct behavior - fail-fast is better than silent failure.
Migration Guide
If ActorSystem fails to start with:
Fix: Grant the application user read permissions to the certificate's private key:
Test Results
Related
Checklist
Next Steps
Optional enhancement (separate PR):