Failed to load OpenSSL provider "legacy": error:07880025:common libcrypto routines::reason(524325)

[Killen911]

Description

I tried to create Kafka producer and got some exceptions.
Firstly I didn't use 'ssl.providers': 'default,legacy' in kafka_config and got:
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: Failed to parse ssl.keystore.location PKCS#12 file: C:\tools\kafka-keystore.p12: error:0308010C:digital envelope routines::unsupported: Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()"}
After I read some posts, added 'ssl.providers' to config and got:

%3|1687798306.488|SSL|kafka-003#producer-1| [thrd:app]: error:12800067:DSO support routines::could not load the shared library: filename(C:\semaphore-agent\vcpkg\packages\openssl_x64-windows\bin\legacy.dll)
%3|1687798306.488|SSL|kafka-003#producer-1| [thrd:app]: error:12800067:DSO support routines::could not load the shared library
Traceback (most recent call last):
File "C:\Projects\Kafka\kfk.py", line 32, in <module>
producer = Producer(kafka_config)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: Failed to load OpenSSL provider "legacy": error:07880025:common libcrypto routines::reason(524325): name=legacy"}

Windows explorer can't find folder C:\semaphore-agent. And I don't know why it use this path.
What can I do to fix and run it?

How to reproduce

kafka_config = {
'acks': 1,
'bootstrap.servers': servers,
'client.id': socket.gethostname(),
'ssl.keystore.location': 'C:\\tools\\kafka-keystore.p12',
'ssl.keystore.password': '12345',
'ssl.key.password': '12345',
'security.protocol': 'SSL',
'ssl.providers': 'default,legacy'
}
producer = Producer(kafka_config)

Checklist

• confluent-kafka-python '2.1.1', 33620224 and librdkafka 2.1.1', 33620479
• Apache Kafka broker version: 2.4.1
• Operating system: Windows 10 x64
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[hadrienbecle]

I have the same problem with confluent-kafka-dotnet. I think the problem comes from librdkafka and their problematic linking of OpenSSL.

C:\semaphore-agent is a folder on the agent that builds librdkafka. I don't know how to properly link/unlink the dependency, so I went around and provided OpenSSL where librdkafka wants it:

1. Install OpenSSL on Windows (I used Chocolatey: choco install openssl)
2. Create the folder C:\semaphore-agent\vcpkg\packages\openssl_x64-windows\
3. Symlink your newly installed OpenSSL: mklink /D "C:\semaphore-agent\vcpkg\packages\openssl_x64-windows\bin\" "C:\Program Files\OpenSSL-Win64\bin\"

It finally works for me. It's dodgy, we shouldn't have to do that, but at least it works...

[malonsocasas]

Same issue with confluent-kafka-python >= 2.1.0 but not with version 2.0.2

Probably related to this specific change: https://github.com/confluentinc/librdkafka/pull/4234/files#diff-a52abbe34637fa09df7a73e3d805baa7392edeb32d7b0c4a85503d7d67a3b015L104-L108

Apparently, no-module option is required to have a static build with "legacy" provider: openssl/openssl#17679

[aentwist]

In OpenSSL 3 (3.9 at least), the legacy provider is not loaded by default. However, it is still there - we just need to enable it. We can do this using the OpenSSL configuration file.

See https://github.com/openssl/openssl/blob/master/README-PROVIDERS.md

EDIT: Ok, Kafka doesn't respect the way OpenSSL 3 is working. In my Docker container I now have,

# openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.0.9
    status: active
  legacy
    name: OpenSSL Legacy Provider
    version: 3.0.9
    status: active

But I still get the error,

%3|1702946439.025|SSL|rdkafka#producer-1| [thrd:app]: error:12800067:DSO support routines::could not load the shared library: filename(/usr/lib64/ossl-modules/legacy.so): /usr/lib64/ossl-modules/legacy.so: cannot open shared object file: No such file or directory
%3|1702946439.025|SSL|rdkafka#producer-1| [thrd:app]: error:12800067:DSO support routines::could not load the shared library
Couldn't create producer
Traceback (most recent call last):
  File "/app/file.py", line 14, in init
    producer = Producer(config)
               ^^^^^^^^^^^^^^^^
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: Failed to load OpenSSL provider "legacy": error:07880025:common libcrypto routines::reason(524325): name=legacy"}

What is going on? librdkafka should have this since v2.

[MSeal]

Need to follow-up on why this wasn't fixed in librdkafka or if it was fixed and the issue was erroneously using the older version