[Bug]: Invalid CSR generated by ESP32s3

[PaulMartinsen]

Contact Details

No response

Version

Jim/ED25519_SHA2_fix

Description

Library creates an invalid certificate signing request on the ESP32s3 when hardware acceleration is enabled. A valid signing request is generated with hardware acceleration disabled, or the code running on a Windows target. Previously I've seen the same problem on the ESP32 and even with hardware acceleration disabled. Have tested on a recent master branch and also ED25519_SHA2_fix. I used @gojimmypi 's branch as he has a lot of fixes for the Espressif implementation of the library.

Output from the signing command when supplied with a good CSR (ESP32s3 without h/w acceleration):

$ openssl.ex
[user_settings.zip](https://github.com/wolfSSL/wolfssl/files/11007185/user_settings.zip)
e x509 -req -in "Input\BLS-001 - ESP32s3 - no HW acceleration - new key - signing request.csr" -days 10   -CA "CertificateAuthority-4096-PublicKey.crt" -CAkey "CertificateAuthority-4096-PrivateKey.pem" -passin pass:secret -CAcreateserial   -extfile "CertificateExtensions.txt" -extensions LumosDevice   -out Output\BLS-001-Device-Certificate.crt -text
Certificate request self-signature ok
subject=C = NZ, L = Hamilton, O = "Blue Leaf Software, Ltd", CN = Lumos, serialNumber = BLS-001

Output from the signing command when supplied with a bad CSR (ESP32s3 with h/w acceleration):

$ openssl.exe x509 -req -in "Input\BLS-001 - ESP32s3 - HW acceleration - New private key - Jim ED25519_SHA2_fix -signing request.csr" -days 10   -CA "CertificateAuthority-4096-PublicKey.crt" -CAkey "CertificateAuthority-4096-PrivateKey.pem" -passin pass:secret -CAcreateserial   -extfile "CertificateExtensions.txt" -extensions LumosDevice   -out Output\BLS-001-Device-Certificate.crt -text
Certificate request self-signature did not match the contents
545B0000:error:0200008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:crypto\rsa\rsa_pk1.c:75:
545B0000:error:02000072:rsa routines:rsa_ossl_public_decrypt:padding check failed:crypto\rsa\rsa_ossl.c:599:
545B0000:error:1C880004:Provider routines:rsa_verify:RSA lib:providers\implementations\signature\rsa_sig.c:774:
545B0000:error:06880006:asn1 encoding routines:ASN1_item_verify_ctx:EVP lib:crypto\asn1\a_verify.c:217:

If I take the private key that's generated on the device I can create a CSR as below (in Windows), but that produces the same error message as above when I try to sign the CSR as above suggesting the problem might be with the private key.
openssl req -new -key "Input\BLS-001 - ESP32s3 - HW acceleration - New private key - Jim ED25519_SHA2_fix.pem" -nodes -out Output\BLS-001-req.csr

Code used to generate a private key and signing request on the device:

bool CCredentialStore::GenerateCertificate(const char* pchSerial, const char* pAddress)
{
  // Generate a new private device key. Using a RSA key because the ESP32 has hardware
  // acceleration for that. 
  // https://www.wolfssl.com/documentation/manuals/wolfssl/chapter07.html#rsa-key-generation
  // https://www.wolfssl.com/documentation/manuals/wolfssl/group__Random.html#function-wc_initrng
  // https://www.wolfssl.com/documentation/manuals/wolfssl/group__RSA.html#function-wc_initrsakey
  // https://www.wolfssl.com/documentation/manuals/wolfssl/group__RSA.html#function-wc_makersakey
  
  // todo: check if this is using the Espressif RNG. If it is, make sure the radio
  // is on. The radio is used as an entropy source to make sure that true random
  // numbers are generated. If it isn't using the Espressif RNG, maybe it should 
  // be since that should generate True Random Numbers. 
  WC_RNG RandomNumberGenerator; 
  int nResult = wc_InitRng(&RandomNumberGenerator);
  if (0 != nResult)
  {
    ReportError("InitRng", nResult);
    return false;
  }

  unique_ptr<RsaKey> pNewKey(new RsaKey());
  nResult = wc_InitRsaKey(pNewKey.get(), nullptr);
  if (0 != nResult)
  {
    ReportError("InitRsaKey", nResult);
    return false;
  }

  nResult = wc_MakeRsaKey(pNewKey.get(), PrivateKeySize, 65537, &RandomNumberGenerator);
  if (0 != nResult)
  {
    ReportError("MakeRsaKey", nResult);
    return false;
  }
  
  const size_t szPrivateKeyBuffer = 2048;
  auto puPrivateKeyDer = make_unique<uint8_t[]>(szPrivateKeyBuffer);
  nResult = wc_RsaKeyToDer(pNewKey.get(), puPrivateKeyDer.get(), szPrivateKeyBuffer);
  if (nResult > 0)
  {
    WriteCertificate(puPrivateKeyDer.get(), nResult, PRIVATEKEY_TYPE);
  }
  else
  {
    cerr << "Error: wc_RsaKeyToDer\n";
  }

  // Generate a certificate signing request. 
  // https://www.wolfssl.com/documentation/manuals/wolfssl/chapter07.html#certificate-signing-request-csr-generation
  // https://www.wolfssl.com/documentation/manuals/wolfssl/asn__public_8h.html#function-wc_makecertreq
  // https://www.wolfssl.com/documentation/manuals/wolfssl/asn__public_8h.html#function-wc_signcert
  unique_ptr<Cert> pRequest(new Cert());
  wc_InitCert(pRequest.get());
  pRequest->daysValid = 30;

#if IS_WINDOWS
  strncpy_s(pRequest->subject.country, "NZ", sizeof(pRequest->subject.country));
  strncpy_s(pRequest->subject.org, "Blue Leaf Software, Ltd", sizeof(pRequest->subject.org));
  strncpy_s(pRequest->subject.locality, "Hamilton", sizeof(pRequest->subject.locality));
  strncpy_s(pRequest->subject.commonName, "Lumos", sizeof(pRequest->subject.commonName));
#else
  strncpy(pRequest->subject.country, "NZ", sizeof(pRequest->subject.country));
  strncpy(pRequest->subject.org, "Blue Leaf Software, Ltd", sizeof(pRequest->subject.org));
  strncpy(pRequest->subject.locality, "Hamilton", sizeof(pRequest->subject.locality));
  strncpy(pRequest->subject.commonName, "Lumos", sizeof(pRequest->subject.commonName));
#endif
  CFixedStringPrint SerialDev(pRequest->subject.serialDev, sizeof(pRequest->subject.serialDev));
  SerialDev << pchSerial;

  const size_t szCertificateRequestBuffer = 4096;
  auto puCertificateRequest = make_unique<uint8_t[]>(szCertificateRequestBuffer);
  int nRequestSize = wc_MakeCertReq(pRequest.get(), puCertificateRequest.get(), szCertificateRequestBuffer, pNewKey.get(), nullptr);
  if (nRequestSize <= 0)
  {
    ReportError("MakeCertReq", nRequestSize);
    return false;
  }

  cout << "SignCert\n";
  pRequest->sigType = CTC_SHA256wRSA;
  int nCertificateSize = wc_SignCert(pRequest->bodySz, pRequest->sigType, puCertificateRequest.get(), szCertificateRequestBuffer, pNewKey.get(), nullptr, &RandomNumberGenerator);
  if (nCertificateSize <= 0)
  {
    ReportError("SignCert", nCertificateSize);
    return false;
  }

  WriteCertificate(puCertificateRequest.get(), nCertificateSize, CERTREQ_TYPE);
  
  cout << "\ndone.\n";

  return true;
}

void CCredentialStore::WriteCertificate(const uint8_t *pDERCertificate, size_t szCertificate, int nType)
{
  const size_t szTextRequestBuffer = 4096;
  auto puTextRequest = make_unique<uint8_t[]>(szTextRequestBuffer);
  memset(puTextRequest.get(), 0, szTextRequestBuffer);
  int nPemSize = wc_DerToPem(pDERCertificate, szCertificate, puTextRequest.get(), szTextRequestBuffer, nType);
  if (nPemSize <= 0 || nPemSize == szTextRequestBuffer)
  {
    ReportError("DerToPem", nPemSize);
    return;
  }

  // make sure null terminated. 
  puTextRequest.get()[nPemSize - 1] = 0;

  cout << reinterpret_cast<char*>(puTextRequest.get()) << endl << endl;
}

Also attached, WolfSSL user_settings.zip files, Keys and certificate signing requests.zip

Reproduction steps

No response

Relevant log output

No response

[dgarske]

@anhu will you review this or assign to another engineer? Jim is traveling next week.

[anhu]

Hi @PaulMartinsen ,

Thank you very much for all the details. I have read your report very carefully and reproduced what you have seen on windows on my linux machine. Her are my steps (for the benefit of future readers):

#create root cert
$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out self_signed_req.pem -nodes
<enter dummy data>
# try to create the cert
$ openssl x509 -req -in BLS-001\ -\ ESP32s3\ -\ HW\ acceleration\ -\ New\ private\ key\ -\ Jim\ ED25519_SHA2_fix\ -signing\ request.csr -CA self_signed_req.pem  -CAkey key.pem -CAcreateserial -out certificate.pem -text 

Interestingly, when I use openssl to generate a CSR with the same private key, the same thing happens:

openssl req -new -key "BLS-001 - ESP32s3 - HW acceleration - New private key - Jim ED25519_SHA2_fix.pem" -nodes -out req.csr
<enter dummy data>
openssl x509 -req -in req.csr -CA self_signed_req.pem  -CAkey key.pem -CAcreateserial -out certificate.pem -text

This would make me agree with you that there is a problem with the private key. However, I find it strange that the problem would manifest itself in the usage of the public key that is based on the private key and not usage of the private key itself.

I will first do my best to better understand what kind of problem OpenSSL is encountering and report my findings here. However, please note that I do not have the ESP hardware. As such, once I report my findings, I will need to assign to another developer who has the relevant hardware.

Warm regards, Anthony

[anhu]

It would appear that OpenSSL is expecting this format of padding:

    /*
     * The format is
     * 00 || 01 || PS || 00 || D
     * PS - padding string, at least 8 bytes of FF
     * D  - data.
     */

I think this is called PKCS1 padding. Instead of getting the initial 0, it is finding a value of decimal 135. So, it would appear the padding is being done incorrectly.

I will need to dig into this a big deeper. @PaulMartinsen , I'm unfamiliar with the ESP hardware acceleration code and hardware. I will need to lookup whether it is the code or the hardware that is responsible for the padding. Please stay tuned while I do the research. If you can provide insights, I'd really appreciate it.

Warm regards, Anthony

[PaulMartinsen]

Hi @anhu , thanks very much for your analysis. A great idea to use the private key to sign a CSR locally.

My understanding is the ESP hardware accelerates low level operations (e.g., Z = X * Y (mod M) ; Z = X^Y mod M), SHA and AES operations. I don't know enough about how the private key's are generated to know where the padding is being done though. I do have TLSv1.3 connections working on the ESP32s3 using the hardware acceleration and @gojimmypi has fixed some issues in SHA (that's the branch I mentioned above).

Is there a function in the library to load a PEM private key? I could upload a private key onto the device then do the CSR on that to confirm the private key is the issue if that were useful. I couldn't find a library function for that though. Another possibility would be to use the Wolf library to make a CSR using the bad private key. Would we expect the problem to be detected by wc_MakeCertReq or is that not possible?

Have a great day
Paul.

[anhu]

Hi @PaulMartinsen ,
The API is wc_RsaPrivateKeyDecode() . It takes as input a buffer containing a DER encoded key. If your key is PEM encoded, you will also need wc_KeyPemToDer(). There are examples of how to use these here: https://github.com/wolfSSL/wolfssl-examples/tree/master/certgen

[PaulMartinsen]

Thanks @anhu .

I modified the code sample from above to load the private key I'd previously made with OpenSSL and loaded onto the ESP32s3 instead of calling wc_MakeRsaKey, copied below. Then the rest of the code created the certificate signing request, with the following output:

$ openssl.exe x509 -req -in "Input\Sign existing certificate - ESP32s3.csr" -days 10   -CA "CertificateAuthority-4096-PublicKey.crt" -CAkey "CertificateAuthority-4096-PrivateKey.pem" -passin pass:secret -CAcreateserial   -extfile "CertificateExtensions.txt" -extensions LumosDevice   -out Output\test-Device-Certificate.crt -text
Certificate request self-signature ok
subject=C = NZ, L = Hamilton, O = "Blue Leaf Software, Ltd", CN = Lumos, favouriteDrink = BLS-001

So that doesn't have the self-signature miss-match I see when using wc_MakeRsaKey to create a new private key, which again suggests the issue lies with the private key itself, somehow. One weird behaviour is that favouriteDrink instead of serialNumber appears in the subject distinguished name. Though the value (BLS-001) is what I loaded for subject.serialDev. That could just be a distraction though.

#if 1
  ifstream file("/content/Prvt+PubKy.crt");
  file.seekg(0, ios::beg);
  ostringstream Buffer;
  Buffer << file.rdbuf();
  string strContent = Buffer.str();
  
  unsigned char abyBuffer[1250];
  int nWritten = wc_KeyPemToDer((const unsigned char*)(strContent.c_str()), strContent.length(), abyBuffer, sizeof(abyBuffer), nullptr);
  if (nWritten < 0)
  {
    ReportError("wc_KeyPemToDer", nWritten);
    return false; 
  }
  
  word32 nStartOfKey = 0; 
  nResult = wc_RsaPrivateKeyDecode(abyBuffer, &nStartOfKey, pNewKey.get(), nWritten);
  if (0 != nResult)
  {
    ReportError("wc_RsaPrivateKeyDecode", nResult);
    return false;
  }
#else
  nResult = wc_MakeRsaKey(pNewKey.get(), PrivateKeySize, 65537, &RandomNumberGenerator);
  if (0 != nResult)
  {
    ReportError("MakeRsaKey", nResult);
    return false;
  }
#endif

[anhu]

Hi @PaulMartinsen ,

Wow. Thank you so much for your prompt reaction time and hard work!! It seems quite clear to me now that when the private key is generated, it has the public components in it. Then, when the CSR is generated, the public components are just being copied into the CSR. Once the public key is then actually used to verify the CSR, the padding problem is discovered.

As you said in your original post, this only happens when ESP32s3 acceleration is enabled. Please forgive my ignorance, but can you please let me know which flag/macro turns on ESP32s3 acceleration?

Warm regards, Anthony

[anhu]

@PaulMartinsen ,

Is this an urgent issue for you? Is temporarily disabling hardware acceleration a viable temporary solution?

Warm regards, Anthony

[anhu]

For the benefit of future readers:

I have looked in the code where CONFIG_IDF_TARGET_ESP32S3 is defined and as it relates to RSA, it seems to affect fairly low level mathematical operations. As such, it would seem to not relate to padding. As I have no hardware, I can't think of any further path I can take. I will confer with my colleagues.

Warm regards, Anthony

[anhu]

Hi @PaulMartinsen ,

The next sensible step would be to turn off hardware acceleration and see the differences in the csr that are generated. Can you try that? I do not have the correct hardware so I can't try that.

Note that @gojimmypi will be helping here to setup an environment to reproduce what you are seeing. However, due to other activities he will have limited time to work on this. I'm assigning this to him, but will continue to monitor and help out on this issue.

[anhu]

Oh, the unaccelerated version is in the zip file. Let me look into that.

[PaulMartinsen]

Hi @anhu

This isn't a blocking problem for me at the moment as I can load externally generated keys onto the device. But I'm keenly interestedin, and happy to help with, its resolution :).

I add the following pre-processor symbols to user_settings.h to turn off hardware acceleration:

#if 1
// Undocumented. Disable hardware acceleration of cryptography functions including RSA
#define NO_ESP32WROOM32_CRYPT

// Undocumented. Disable hardware acceleration of RSA primitives?
#define NO_WOLFSSL_ESP32WROOM32_CRYPT_RSA_PRI 

// Undocumented. Disable hardware acceleration of SHA (e.g., SHA-256, SHA-512)
#define NO_WOLFSSL_ESP32WROOM32_CRYPT_HASH

// Undocumented. Disable hardware acceleration of AES
#define NO_WOLFSSL_ESP32WROOM32_CRYPT_AES

#endif

BLS-001 - ESP32s3 - no HW acceleration - new key - signing request.csr in Keys.and.certificates.zip was generated with hardware acceleration off.

BTW, I just tried openssl rsa -in "BLS-001 - ESP32s3 - HW acceleration - New private key - Jim ED25519_SHA2_fix.pem" -text -pubout to see if any errors but it printed private key and public key things without reporting any errors.

I'm not quite sure what you mean by comparing the CSR with and without hardware acceleration. I'm generating a separate private key each time. Since you saw the same problem with OpenSSL, I'm not sure if reusing the hardware acceleration generated private key in a non-hardware accelerated CSR will tell us anything new? Did I missunderstand?

Thanks for your help with this. Much appreciated.

BTW, I'm in New Zealand. We are about 12 hours behind GMT, so sometimes there might be delays in my responses :)