Azure Kubernetes `publick8s` suffers from SNAT port exhaustion: network slowness

[dduportal]

Service(s)

Azure

Summary

The AKS cluster publick8ssuffers from SNAT port exhaustion since around 1 month (example below for the last 24 hours):

It causes the following problems:

• Pagerduty is spamming us with false positive alerts reporting unusual response time from repo.jenkins-ci.org and reports.jenkins.io services while other probes (on other clusters) are reporting valid response times.
• The rest of the public network (including ci.jenkins.io and its agents, even on the peered network on the sponsored subscription,) is slowed down:
  • ci.jenkins.io jobs on Windows agents are much slower than 21 days ago #3904 (even if the network slowness is not the only reason)
  • Mirrorbits got pods restart on publick8s #3799 (mirrorbit pods are failing when trying to scan mirrors)

Reproduction steps

No response

[dduportal]

• This problem rings a bell in my memory: https://github.com/jenkins-infra/documentation/blob/5971469e62d93184ad4ffa34b969f53a1c7af45c/archives/aks-1.19.md?plain=1#L101 => we did had and solved it on the former prodpublick8s (replaced by publick8s since then).
  • We could check the method we used (allocating a public IP pool prefix to the loadBlancer used for outbound connectivity)
  • But this time let's track it and even better: as publick8s is terraformed, let's see if we can use config from https://www.danielstechblog.io/detecting-snat-port-exhaustion-on-azure-kubernetes-service/
  • AKS clusters now support NAT gateway outbound method but we cannot migrate an existing AKS cluster to it: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections
  • WiP: reading https://www.danielstechblog.io/preventing-snat-port-exhaustion-on-azure-kubernetes-service-with-virtual-network-nat/ to understand better the concept of Virtual Network NAT which is distinct from a "User NAT Gateway"

[dduportal]

• The ci.jenkins.io networks could be moved out of the publick8s network to separate outbounds routes:
  • Since agents are using the new subscription, we could set it up) to use a NAT gateway for outbound like we already did with trusted.ci and cert.ci
  • ci.jenkins.io is currently using public Ip outbound: moving its controller VM/disk to the secondary subscription would ensure:
    • Complete removal of the public vnet peering (and separate concerns) as ci.j would be on another network
    • More usage on the subscription billing (~450$ monthly)