SSLPeerUnverifiedException Hostname fd33:1a73:fa8f::1 not verified after upgrade to Boot 3.4.0 and Cloud 2024.0.0

[mle-idealo]

Describe the bug
After the upgrade from Spring Boot 3.3.6 to 3.4.0 and Spring Cloud 2023.0.4 to 2024.0.0 we're experiencing issues regarding hostname verification which read like

javax.net.ssl.SSLPeerUnverifiedException: Hostname fd33:1a73:fa8f::1 not verified:
    certificate: sha256/bLcj0Q+HP/EF+4njk0xrQvqb/KtOHnZa2xf+rl9ldkc=
    DN: CN=kube-apiserver
    subjectAltNames: [fd33:1a73:fa8f:0:0:0:0:1, 2a05:d014:396:cd05:0:0:0:e781, 172.16.98.175, 55c2d4e83b3377534d8c22d619c3cb94.gr7.eu-central-1.eks.amazonaws.com, ip-172-16-98-175.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:334)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
        [...]
	at java.lang.Thread.run(Thread.java:1570)

Wrapped by: java.io.IOException: Hostname fd33:1a73:fa8f::1 not verified:
    certificate: sha256/bLcj0Q+HP/EF+4njk0xrQvqb/KtOHnZa2xf+rl9ldkc=

    DN: CN=kube-apiserver
    subjectAltNames: [fd33:1a73:fa8f:0:0:0:0:1, 2a05:d014:396:cd05:0:0:0:e781, 172.16.98.175, 55c2d4e83b3377534d8c22d619c3cb94.gr7.eu-central-1.eks.amazonaws.com, ip-172-16-98-175.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]

	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:792)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:193)
	... 20 common frames omitted

Wrapped by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [offer-attribute-assignor-6778d89688-pdm8h]  in namespace: [offer-attribute-assignor]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:195)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:149)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.isReady(BaseOperation.java:919)
	... 5 frames excluded
	... 13 common frames omitted

Wrapped by: org.springframework.context.ApplicationContextException: Failed to start bean 'leaderInitiator'
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:326)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:510)
	at java.lang.Iterable.forEach(Iterable.java:75)
	... 10 frames excluded
	at de.idealo.orca.attribute.ApplicationKt.main(Application.kt:24)

There are various bug reports similar to this one available but they all date years back and are already closed.
I suspect the crucial change anywhere within io.fabric8:kubernetes-client between versions 6.9.2 (Cloud 2023.0.4) and 6.13.4 (Cloud 2024.0.0).

From my understanding the address fd33:1a73:fa8f::1 should be fine regarding verification.
Did we miss something?

[ryanjbaxter]

Can you try setting the environment variable KUBERNETES_DISABLE_HOSTNAME_VERIFICATION_SYSTEM_PROPERTY or setting the system property kubernetes.disable.hostname.verification to true?

I am pretty sure this change altered the behavior.
fabric8io/kubernetes-client@08b0e9f#diff-7506ddc44edef04cb34dcde13c56d00ed3b1d35d848ded8f1924ed44c03db953R403

[mle-idealo]

Unfortunately despite set env variable, no change in behavior. The host name is still getting verified:

"stack_trace":"javax.net.ssl.SSLPeerUnverifiedException: Hostname fd33:1a73:fa8f::1 not verified
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:334)

I've double-checked this also by setting the respective system property instead of the env var, same result, host name is still getting verified.

[ryanjbaxter]

Sorry looks like the environment variable should be KUBERNETES_DISABLE_HOSTNAME_VERIFICATION. Can you provide a sample that reproduces the issue?

[mle-idealo]

Unfortunately I am off until Monday but regarding the env var, one of my colleagues, @andibraeu, could give this a try. However apart from the slightly renamed env var name, should not also the system property you mentioned yield the same effect?

Regarding the sample, I could provide one next week. I'll just shrink our project down, since this popped up in many repos and they're not that fancy, just some Kafka Listeners and a MongoDb.

[wind57]

It could actually be us here... The code used to do:

Inet4Address.getLocalHost().getHostName();

and now it does:

InetAddress.getLocalHost().getHostName();

I don't know this API too much, but I will do some reading and come back

off-top: hats off for using Shenandoah! :) we use it also in production for quite some time

[wind57]

It's a bit weird that you do not have the env variable HOSTNAME in the pod btw. can you issue an env inside the pod? because our code does this:

	public static String hostName() throws UnknownHostException {
		String hostName = EnvReader.getEnv(HOSTNAME);
		if (StringUtils.hasText(hostName)) {
			return hostName;
		}
		else {
			return InetAddress.getLocalHost().getHostName();
		}
	}