bug(containerd): Pod creation fails consistently: failed to create containerd task: failed to create shim task: OCI runtime create failed: chown /dev/stdout: permission denied: unknown

[liviobue]

What happened:
This issue is similar to issue 2366 (can be merged if possible)

When applying the provided manifest with an initContainer specifying a non-root user (runAsUser: 1337) and restrictive securityContext settings (privileged: false, allowPrivilegeEscalation: false), the Pod fails to start the initContainer. The error occurs during container initialization with the message:

unable to start container process: error during container init: unable to setup user: chown /dev/stdout: permission denied: unknown.

Running the initContainer as root (runAsUser: 0) bypasses this problem, and the Pod starts successfully.

What you expected to happen:
The Pod should start the initContainer with the specified non-root user and securityContext settings without encountering permission denied errors, allowing the container initialization to complete successfully.

How to reproduce it:

1. Apply the following Pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: init-seccontext-example
  namespace: default
spec:
  initContainers:
  - name: privileged-init
    image: busybox
    command:
    - sh
    - -c
    - |
      echo "Hello"
    securityContext:
      runAsUser: 1337
      privileged: false
      allowPrivilegeEscalation: false
  containers:
  - name: main-container
    image: busybox
    command:
    - sh
    - -c
    - |
      echo "Running main container"; sleep 3600
    securityContext:
      runAsUser: 0
      allowPrivilegeEscalation: true
      privileged: true
      runAsNonRoot: false
  restartPolicy: Never
  securityContext:
    seccompProfile:
      type: RuntimeDefault

2. Observe the following Pod events indicating failure:

Type     Reason     Age    From               Message
----     ------     ----   ----               -------
Normal   Scheduled  5m14s  default-scheduler  Successfully assigned default/init-seccontext-example to ip-x.x.x.x.eu-central-2.compute.internal
Normal   Pulling    5m13s  kubelet            Pulling image "busybox"
Normal   Pulled     5m12s  kubelet            Successfully pulled image "busybox" in 784ms (784ms including waiting). Image size: 2223685 bytes.
Normal   Created    5m12s  kubelet            Created container: privileged-init
Warning  Failed     5m12s  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to setup user: chown /dev/stdout: permission denied: unknown

3. Modify the initContainer's runAsUser to 0 (root). (remove also: privileged: false, allowPrivilegeEscalation: false)

4. Reapply the manifest and observe the Pod starts successfully without the error.

Environment:

• AWS Region: eu-central-2
• Instance Type(s): t3.xlarge
• Cluster Kubernetes version: v1.32.7-eks-ace6451
• Node Kubernetes version: v1.32.7-eks-3abbec1
• AMI Version: amazon-eks-node-al2023-x86_64-standard-1.32-v20250813

[cartermckinnon]

Do you mean this worked previously, and now doesn’t?

[liviobue]

Yes, it previously worked. We’re running it on another EKS cluster with the same Kubernetes version (1.32), same node type, and same region.

The only difference I can see so far is the AMI version. I believe it was working with AMI Release v20250627, but I’ll double-check that tomorrow to confirm.

[cartermckinnon]

I'm not able to reproduce this on v20250813. The version of runc changed between v20250704 and v20250715, so I tried those as well -- no issue.

Are you doing anything custom in your user data?

What I tried

Cluster:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: github-issue-2385
  region: us-west-2
  version: '1.32'

nodeGroups:
  - name: nodes-0704
    minSize: 1
    maxSize: 1
    desiredCapacity: 1
    amiFamily: AmazonLinux2023
    instanceType: m5.large
    ami: ami-01e47de4ce1b43437
  - name: nodes-0715
    minSize: 1
    maxSize: 1
    desiredCapacity: 1
    amiFamily: AmazonLinux2023
    instanceType: m5.large
    ami: ami-0100a64ae7193f8c8
  - name: nodes-0813
    minSize: 1
    maxSize: 1
    desiredCapacity: 1
    amiFamily: AmazonLinux2023
    instanceType: m5.large
    ami: ami-093c1ecf503bf0d7f

[liviobue]

I double-checked the user data of the nodes. It’s the default EKS node config and I didn’t add any custom bootstrap logic.
This is the full user-data that is being used:

User-data

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    apiServerEndpoint: https://xxxxxxxxxxxxxxxxx.xxx.eu-central-2.eks.amazonaws.com
    certificateAuthority: xxxxxxxxxx
    cidr: 172.20.0.0/16
    name: test-cluster
  kubelet:
    config:
      maxPods: 17
      clusterDNS:
      - 172.20.0.10
    flags:
    - "--node-labels=eks.amazonaws.com/nodegroup-image=ami-07a8558d2b8b13d0f,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=ondemands-2b-2025081907523988590000001f"

--//--

[cartermckinnon]

Are you running any kind of security agent on your nodes?

@ndbaker1 can you sanity check my repro?

[liviobue]

I’ve just recreated the cluster from scratch and made sure no security agent or additional daemon was deployed.
I’m using EKS managed node groups, so I don’t have direct SSH access to the nodes and therefore can’t easily shell into them to check for running security processes.

Additional information:
After recreating the cluster, the pod creation using the provided manifest now fails only on nodes located in AZ eu-central-2b and eu-central-2c. Nodes in eu-central-2a start the Pod successfully.
I’ll keep you updated if I identify any more useful information.

[cartermckinnon]

You could use SSM to get a shell session on an instance and poke around 👍

I wouldn’t expect an issue like this to differ by AZ if the AMI is the problem. Do you have any kind of automation that would try to update/patch software on your instances?

It’d be helpful to see some logs from your nodes, so please open an AWS support case if you can.

[ndbaker1]

@ndbaker1 can you sanity check my repro?

I also can't seem to get any hits so far. tested both us-west-2 (amis in #2385 (comment)) and specifically launching instances into eu-central-2b w/ ami-07a8558d2b8b13d0f.