Cannot run proxy sidecar in an environment with hard ResourceQuotas enabled #1441
Description
Describe the bug
The sidecar is entirely unusable with a namespace where a ResourceQuota with hard limits has been enabled. The webhook does not set or have any capabilities to set such limits as far as I can see from the code, making it impossible to start a process where quotas are enforced.
The code I'm referring to here:
- https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L190
- https://github.com/Azure/azure-workload-identity/blob/main/pkg/webhook/webhook.go#L220
Steps To Reproduce
Create a namespace and apply a ResourceQuota to it, for example:
apiVersion: v1
kind: ResourceQuota
metadata:
name: resource-quota-name
namespace: resource-quota-namespace
spec:
hard:
count/configmaps: "100"
count/cronjobs.batch: "10"
count/deployments.apps: "100"
count/jobs.batch: "100"
count/persistentvolumeclaims: "10"
count/pods: "200"
count/services: "100"
limits.memory: 8Gi
requests.cpu: "6"
requests.memory: 8GiFinally, deploy a workload identity-enabled pod with resources set to this namespace, with the proxy enable using azure.workload.identity/inject-proxy-sidecar: 'true'. It should fail to start and complain about quota errors as mentioned below.
Expected behavior
The proxy and init container will start without issues.
Logs
message: 'pods "<podName>" is forbidden: failed quota:
<resourceQuotaName>: must specify limits.memory for: azwi-proxy,azwi-proxy-init; requests.cpu
for: azwi-proxy,azwi-proxy-init; requests.memory for: azwi-proxy,azwi-proxy-init'
Environment
- Kubernetes version (use
kubectl version): v1.29.2 - Cloud provider or hardware configuration: AKS
- OS (e.g:
cat /etc/os-release):
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
- Kernel (e.g.
uname -a):Linux <nodeName> 5.15.0-1068-azure #77-Ubuntu SMP Fri Jun 21 21:21:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux - Install tools: Terraform, ArgoCD, custom operators
- Network plugin and version (if this is a network-related bug): n/a
- Others: n/a
Additional context
Istio accomplishes a similar end using the proxyCPU and proxyMemory settings via annotations passed in on the pod - perhaps such an approach is applicable here?
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPU
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyCPULimit
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemory
- https://istio.io/latest/docs/reference/config/annotations/#SidecarProxyMemoryLimit