KubernetesClientException - quarkus.kubernetes-client.trust-certs=true not working on Windows

[haraldatbmw]

Describe the bug
OpenShift deployment (quarkus-openshift extension) fails on Windows 10 because of self-signed certificate. The property quarkus.kubernetes-client.trust-certs=true has no effect. The same source code works fine on Linux.

Expected behavior
Deployment to OpenShift.

Workaround
setting environment variable explicit:

set KUBERNETES_TRUST_CERTIFICATES=true
mvn clean package -Dquarkus.kubernetes.deploy=true

Actual behavior
KubernetesClientException

...
[INFO] --- quarkus-maven-plugin:1.5.0.CR1:build (default) @ quarkus-openshift-deployment ---
[INFO] [org.jboss.threads] JBoss Threads version 3.1.1.Final
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building thin jar: C:\git\Quarkus\quarkus-openshift-deployment\target\quarkus-openshift-deployment-1.0.0-SNAPSHOT-runner.jar
[WARNING] [io.quarkus.kubernetes.deployment.KubernetesProcessor] No registry was set for the container image, so 'ImagePullPolicy' is being force-set to 'IfNotPresent'.
[INFO] Checking for existing resources in: C:\git\Quarkus\quarkus-openshift-deployment\src\main\kubernetes.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesProcessor] Generated the Kubernetes manifests: '\openshift.json,\openshift.yml' in 'C:\git\Quarkus\quarkus-openshift-deployment\target\kubernetes'
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeploy] Kubernetes API Server at 'https://xxxx.bmwgroup.net:8443/' successfully contacted.
[INFO] [io.quarkus.container.image.s2i.deployment.S2iProcessor] Performing s2i binary build with jar on server: https://xxxx.bmwgroup.net:8443/ in namespace:harald.
[INFO] [io.quarkus.container.image.s2i.deployment.S2iProcessor] Found: ImageStream openjdk-11 repository: registry.access.redhat.com/ubi8/openjdk-11
[INFO] [io.quarkus.container.image.s2i.deployment.S2iProcessor] Applied: ImageStream quarkus-openshift-deployment
[INFO] [io.quarkus.container.image.s2i.deployment.S2iProcessor] Applied: BuildConfig quarkus-openshift-deployment
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  33.329 s
[INFO] Finished at: 2020-05-26T15:02:34+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.5.0.CR1:build (default) on project quarkus-openshift-deployment: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]         [error]: Build step io.quarkus.container.image.s2i.deployment.S2iProcessor#s2iBuildFromJar threw an exception: io.dekorate.deps.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [ImageStreamTag]  with name: [openjdk-11:latest]  in namespace: [harald]  failed.
[ERROR]         at io.dekorate.deps.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
[ERROR]         at io.dekorate.deps.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:225)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:168)
[ERROR]         at io.dekorate.s2i.util.S2iUtils.waitForImageStreamTags(S2iUtils.java:67)
[ERROR]         at io.quarkus.container.image.s2i.deployment.S2iProcessor.applyS2iResources(S2iProcessor.java:305)
[ERROR]         at io.quarkus.container.image.s2i.deployment.S2iProcessor.createContainerImage(S2iProcessor.java:265)
[ERROR]         at io.quarkus.container.image.s2i.deployment.S2iProcessor.s2iBuildFromJar(S2iProcessor.java:197)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]         at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:932)
[ERROR]         at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
[ERROR]         at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
[ERROR]         at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]         at org.jboss.threads.JBossThread.run(JBossThread.java:479)
[ERROR] Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[ERROR]         at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
[ERROR]         at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
[ERROR]         at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
[ERROR]         at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
[ERROR]         at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645)
[ERROR]         at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
[ERROR]         at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
[ERROR]         at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
[ERROR]         at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
[ERROR]         at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
[ERROR]         at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177)
[ERROR]         at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
[ERROR]         at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151)
[ERROR]         at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062)
[ERROR]         at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:320)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
[ERROR]         at io.dekorate.deps.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]         at io.dekorate.deps.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]         at io.dekorate.deps.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]         at io.dekorate.deps.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]         at io.dekorate.deps.openshift.client.internal.OpenShiftOAuthInterceptor.intercept(OpenShiftOAuthInterceptor.java:69)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
[ERROR]         at io.dekorate.deps.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
[ERROR]         at io.dekorate.deps.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
[ERROR]         at io.dekorate.deps.okhttp3.RealCall.execute(RealCall.java:93)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:395)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:376)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:845)
[ERROR]         at io.dekorate.deps.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:214)
[ERROR]         ... 17 more
[ERROR] Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[ERROR]         at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
[ERROR]         at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
[ERROR]         at java.base/sun.security.validator.Validator.validate(Validator.java:264)
[ERROR]         at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
[ERROR]         at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
[ERROR]         at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
[ERROR]         at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:629)
[ERROR]         ... 61 more
[ERROR] Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[ERROR]         at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
[ERROR]         at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
[ERROR]         at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
[ERROR]         at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
[ERROR]         ... 67 more
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

To Reproduce
Steps to reproduce the behavior:

1. Clone repo https://github.com/haraldatbmw/quarkus-openshift-deployment
2. Login to your OpenShift cluster oc login ...
3. Start deployment mvn clean package -Dquarkus.kubernetes.deploy=true

Configuration

quarkus.native.container-build=true
quarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-native-image:20.0.0-java11

quarkus.s2i.base-jvm-image=registry.access.redhat.com/ubi8/openjdk-11
quarkus.s2i.base-native-image=quay.io/quarkus/ubi-quarkus-native-binary-s2i:20.0.0
quarkus.s2i.native-arguments=-Xms8M -Xmx8M -Xmn8M

quarkus.kubernetes-client.trust-certs=true
quarkus.kubernetes.deployment-target=openshift

quarkus.openshift.expose=true
quarkus.openshift.labels.app=quarkus-demo

Environment (please complete the following information):

• Output of uname -a or ver: Microsoft Windows [Version 10.0.17134.1425]

• Output of java -version:
  openjdk version "11.0.6" 2020-01-14 LTS
  OpenJDK Runtime Environment Zulu11.37+17-CA (build 11.0.6+10-LTS)
  OpenJDK 64-Bit Server VM Zulu11.37+17-CA (build 11.0.6+10-LTS, mixed mode)

• Quarkus version or git rev: 1.5.0.CR1

• Build tool (ie. output of mvnw --version or gradlew --version): Apache Maven 3.6.3

• oc tool:
  oc v3.11.0+0cbc58b
  kubernetes v1.11.0+d4cacc0
  features: Basic-Auth SSPI Kerberos SPNEGO

[geoand]

@haraldatbmw do all other configuration values work as expected on Windows?

[haraldatbmw]

@geoand Yes, all other properties inside the application.properties file works fine.

[geoand]

Thanks. Really weird...

[haraldatbmw]

@geoand Does the quarkus extension set this environment varialbe KUBERNETES_TRUST_CERTIFICATES=true??

[geoand]

I see what the problem is. I'll have a fix soon.

KUBERNETES_TRUST_CERTIFICATES works because the Kubernetes Client always check it

[haraldatbmw]

@geoand weird is that the property quarkus.kubernetes-client.trust-certs=true works in my Linux environment fine. If I set it to false i get the same error on Linux too.

[geoand]

Yeah, it;s just plain inconsistent now, but the fix is easy at least :)

[geoand]

@haraldatbmw would you be willing to check with #9610 ?

[haraldatbmw]

@geoand I will build your branch and check it.

[geoand]

Thanks!

[haraldatbmw]

@geoand So after hours of building over 600 quarkus extensions ;-) I can confirm that the bug is fixed in your branch (https://github.com/geoand/quarkus/tree/%239605).