Docker images with non-root account fails to read token file

[serialx]

What happened:
Tried to install External DNS to my EKS cluster with amazon-eks-pod-identity-webhook

What you expected to happen:
External DNS working with IAM credentials provided by amazon-eks-pod-identity-webhook

How to reproduce it (as minimally and precisely as possible):

1. Enable IAM Roles for Service Accounts on your Cluster
2. Setup amazon-eks-pod-identity-webhook to the cluster.
3. Use the latest master branch of External DNS
4. Annotate the external-dns service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: external-dns
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<REDACTED>:role/K8sExternalDNSRole

5. External DNS pod logs show:

time="2019-09-11T07:31:53Z" level=error msg="WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied"

Note: I confirmed amazon-eks-pod-identity-webhook is working properly when using root accounts in the pods.

Anything else we need to know?:
I've reported this issue to external dns project: kubernetes-sigs/external-dns#1185
And it seems odd that kubernetes service account volume mounted token files are with permission 0644 and the EKS tokens are with permission 0600.

EKS tokens seems to be 0600:

~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/
total 0
drwxrwxrwt    3 root     root           100 Sep 11 06:40 .
drwxr-xr-x    3 root     root            28 Sep 11 06:40 ..
drwxr-xr-x    2 root     root            60 Sep 11 06:40 ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            31 Sep 11 06:40 ..data -> ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            12 Sep 11 06:40 token -> ..data/token
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw-------    1 root     root          1028 Sep 11 06:40 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token

Kubernetes tokens seems to be 0644:

/run/secrets/kubernetes.io/serviceaccount $ ls -al
total 0
drwxrwxrwt    3 root     root           140 Sep 11 07:31 .
drwxr-xr-x    3 root     root            28 Sep 11 07:31 ..
drwxr-xr-x    2 root     root           100 Sep 11 07:31 ..2019_09_11_07_31_49.116680770
lrwxrwxrwx    1 root     root            31 Sep 11 07:31 ..data -> ..2019_09_11_07_31_49.116680770
lrwxrwxrwx    1 root     root            13 Sep 11 07:31 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Sep 11 07:31 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Sep 11 07:31 token -> ..data/token
/run/secrets/kubernetes.io/serviceaccount $ ls -al ..data/
total 12
drwxr-xr-x    2 root     root           100 Sep 11 07:31 .
drwxrwxrwt    3 root     root           140 Sep 11 07:31 ..
-rw-r--r--    1 root     root          1025 Sep 11 07:31 ca.crt
-rw-r--r--    1 root     root            12 Sep 11 07:31 namespace
-rw-r--r--    1 root     root           875 Sep 11 07:31 token

I've seen the code, and the code doesn't specify the token file permission. So the default 0644 should be used. Maybe this is a k8s upstream bug?

Environment:

• AWS Region: ap-northeast-1
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): "eks.1"
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): "1.14"
• Webhook Version: master (Commit ID: d2d6039)

[alfredkrohmer]

Problem seems to be here:
https://github.com/kubernetes/kubernetes/blob/cf76868b3430be015cc1e8443a216bf863a9b9f7/pkg/volume/projected/projected.go#L357

For serviceAccountToken, the mode is hardcoded to octal 0600.

[alfredkrohmer]

See kubernetes/kubernetes#82573

[micahhausler]

Hey thanks for this report, we'll leave this open and resolve once Kubernetes fixes the access mode

[siwyd]

A pity this kind of blocks the usage of the native IAM integration until fixed. Any workable solutions until this is fixed upstream?

[alfredkrohmer]

Seems like this is kind-of intended.

There is a workaround here:
kubernetes-sigs/external-dns#1185 (comment)

[serialx]

Adding securityContext to the yaml file resolved this issue. Updated the PR to update the AWS tutorial:
kubernetes-sigs/external-dns#1185

[micahhausler]

I see a lot of people cross-referencing issues to this. For anyone else stumbling across this, can you reference the upstream issue too? (kubernetes/kubernetes#82573) That will give more signal to upstream that this is a painful issue