Theoretical maximum number of Knative Services for a single cluster

[rhuss]

Ask your question here:

According to Kubernetes Scalability thresholds there is an upper limit for the number of Kubernetes Services that is based on the maximum number of iptable entries on a node. Currently, this limit is at 10000 services (if I understand correctly, this is independent of the number of nodes in a cluster since every node needs to have the same iptables routing).

Since every Knative service translates at a minimum to 3 Kubernetes services (1 ExternalName service pointing to the ingress gateway, and 2 services for each revision (public/private), the theoretical maximum of Knative Services in a cluster would be ~ 3350 Knative services (and much less if using multiple revisions and/or other workloads on this cluster beside Knative).

My questions would be:

• Is this assumption (3350 KServices per cluster) a fair one ?
• Isn't this limitation very severe for a serverless model where potentially many KService is scaled to zero but still occupy iptable entries? So it could be that you have an empty cluster (everything scaled to zero), but your routing tables are still exhausted ?
• Would it be a desiarable goal to reduce the number of Kubernetes Services attached to a Knative Service? (like down to 1:1) ?
• Is there a way to circumvent the iptables limitation? (by maybe using a different tech for service mapping)

[rhuss]

For reference, I found this very interesting blog post which points to this presentation on how to Scaling Kubernetes to Support 50000 Services. It uses IPVS. No idea whether IPVS is supported by Kubernetes out of the box.

[psschwei]

I think it is (though I don't have any experience with it): https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs

[porcelli]

This is a significant limitation for the Knative serverless use case. If you consider that in a serverless environment, you may have ~5% of registered services active, you end up with 95% of the cluster idle.
I would love to hear how others experience scale with the Knative serverless.

[ggaaooppeenngg]

In some environments, the clusters are initialized with limit service CIDR besides the limit of iptables. The number of services is even smaller than 10000. If we support headless services, we can get rid of this limitation. I propose to offer an option in the configuration to turn on headless in order to not assign an ip to service.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[rhuss]

/remove-lifecycle stale