Re-enable support for IAM Roles for Service Accounts

[hasheddan]

What happened?

To conform with Crossplane's new default security context for stacks (crossplane/crossplane#1444), the provider-aws container now runs as non-root (#202). Because we set the non-root user in the Dockerfile, users must rebuild the container to run as root user in order to be able to read the AWS credentials that are injected from the service account into /var/run/secrets/ in the container.

How can we fix it?

There are a few immediate options I could see here:

1. Create a separate release for running in "insecure mode". This could use a workaround like Fix AWS IAM Roles for Service Accounts permission problem. kubernetes-sigs/external-dns#1185 (comment).
2. Add fields on ClusterStackInstall (or the next iteration of the installation unit). This seems like the most sustainable long-term solution as there is desire to move away from including an install.yaml in provider packages (install.yaml allows a full Deployment spec to be declared which can be problematic crossplane/crossplane#1441).
3. Document how a user may build their own package from source in AUTHENTICATION.md. This is a good short-term solution.

[hasheddan]

Some other relevant issues/PRs:

• Docker images with non-root account fails to read token file aws/amazon-eks-pod-identity-webhook#8
• Projected serviceAccountToken has mode of mounted file hardcoded to 0600 kubernetes/kubernetes#82573
• proposal for file permission handling in projected service account volume kubernetes/enhancements#1598

[muvaf]

@hasheddan Looking at kubernetes-sigs/external-dns#1185 (comment) , is there any unexpected or bad side effects of using securityContext.fsGroup: 65534 for all packages in package-manager?

[hasheddan]

@muvaf I think that would be fine. Another option would be to just set it in the install.yaml for AWS and then run Crossplane with —insecure-pass-full-deployment, which would cause Crossplane to not overwrite the security context. That being said, it would be nice if this “just worked”, so I think setting it in Crossplane should be the right solution 👍 I’ll get this fixed!