fix(librdkafka): Copy certs bundle to path expected by librdkafka v2.3.0

librdkafka uses libcurl for fetching OIDC tokens. However, depending on OS, the
location where the certs ca-bundle for validating the tls connection between
librdkafka and the auth provider is searched differs. We now copy the ca cert
bundle to multiple paths to make sure it is found.

While this is a bit hacky, there are limited alternatives we can take because:

- until now (with librdkafka 1.9.2) those certs were searched under
/etc/ssl/certs/ca-certificates.crt but after the update they are searched
under /etc/pki/tls/certs/ca-bundle.crt

- libcurl doesn't get passed/doesn't respect the caCert location
or .pem file set via librdkafka/kafka configs
(see confluentinc/librdkafka#375).

- there is no way clear way to influence the libcurl search path via env vars
(setting the CURL_CA_BUNDLE env var has no effect), with search paths fully
decided at compile-time.

The likely chain of reasons this has popped-up when updating librdkafka to
v2.3.0:
  - we're building {model, pipeline}gateway docker images based on debian
  bullseye image
  - we're copying the build results into redhat ubi9 container
  - libcurl was statically built in previous librdkafka, so was
  using the debian search paths on ubi9
  - libcurl ends up being dynamically linked in the current librdkafka
  (to ubi9 version) or has changed its default ca bundle search path

Author: lc525

scheduler/Dockerfile.modelgateway

@@ -13,8 +13,9 @@ FROM registry.access.redhat.com/ubi9/ubi-minimal as certs
 # Kafka dependencies necessitate leaving CGo enabled and using a base image with C dependencies
 FROM registry.access.redhat.com/ubi9/ubi-micro:9.3-13
 
-# Kafka OIDC token retriva certs: https://github.com/confluentinc/librdkafka/issues/3751
+# Kafka OIDC token retrieve certs (librdkafka using curl): https://github.com/confluentinc/librdkafka/issues/3751
 COPY --from=certs /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
+COPY --from=certs /etc/ssl/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.crt
 
 # Broker Certificates
 COPY --from=certs /etc/ssl/certs/ca-bundle.crt /tmp/certs/kafka/broker/ca.crt

scheduler/Dockerfile.pipelinegateway

@@ -13,8 +13,9 @@ FROM registry.access.redhat.com/ubi9/ubi-minimal as certs
 # Kafka dependencies necessitate leaving CGo enabled and using a base image with C dependencies
 FROM registry.access.redhat.com/ubi9/ubi-micro:9.3-13
 
-# Kafka OIDC token retriva certs: https://github.com/confluentinc/librdkafka/issues/3751
+# Kafka OIDC token retrieve certs (librdkafka using curl): https://github.com/confluentinc/librdkafka/issues/3751
 COPY --from=certs /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
+COPY --from=certs /etc/ssl/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.crt
 
 # Broker Certificates
 COPY --from=certs /etc/ssl/certs/ca-bundle.crt /tmp/certs/kafka/broker/ca.crt