Unable to Use amazon-ecr-credential-helper with EKS pod identity.

[cwichka]

With IRSA configuration, i was able to properly use the amazon ecr credential helper to grant a kubernetes pod access the push and pull from ECR.

When i wanted to switch to EKS pod identity instead of IRSA, i was not able to access ECR, and i keep getting this error while pushing the image to ECR

#13 exporting config sha256:68f2ebf6de8cb5137e2b3447dd5df91ca21f5dadfe6e29ec137929f9a4465b58 done
panic: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc0000dd39d?, 0xfd1?})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x225
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x867e20?, 0xa9e6b8?}}, {0xc0000c2600, 0x2f})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:63 +0x28d
github.com/docker/docker-credential-helpers/credentials.Get({0x8678e8, 0xc000093bc0}, {0x8643c0?, 0xc0000a2000?}, {0x8643e0, 0xc0000a2008})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x20f
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x8678e8?, 0xc000093bc0?}, {0x7ffdabf5b6ef?, 0xc0000cdef0?}, {0x8643c0?, 0xc0000a2000?}, {0x8643e0?, 0xc0000a2008?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x77
github.com/docker/docker-credential-helpers/credentials.Serve({0x8678e8?, 0xc000093bc0?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xf3
main.main()
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:45 +0x194
panic: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc0000dd39d?, 0xfd1?})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x225
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x867e20?, 0xa9e6b8?}}, {0xc0000c2600, 0x2f})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:63 +0x28d
github.com/docker/docker-credential-helpers/credentials.Get({0x8678e8, 0xc000093bc0}, {0x8643c0?, 0xc0000a2000?}, {0x8643e0, 0xc0000a2008})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x20f
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x8678e8?, 0xc000093bc0?}, {0x7ffe880696ef?, 0xc0000cdef0?}, {0x8643c0?, 0xc0000a2000?}, {0x8643e0?, 0xc0000a2008?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x77
github.com/docker/docker-credential-helpers/credentials.Serve({0x8678e8?, 0xc000093bc0?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xf3
main.main()
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:45 +0x194
panic: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc00002b51d?, 0xfd1?})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x225
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x867e20?, 0xa9e6b8?}}, {0xc00002e8d0, 0x2f})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:63 +0x28d
github.com/docker/docker-credential-helpers/credentials.Get({0x8678e8, 0xc00003bc00}, {0x8643c0?, 0xc000014010?}, {0x8643e0, 0xc000014018})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x20f
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x8678e8?, 0xc00003bc00?}, {0x7ffc8af3f6ef?, 0xc00012fef0?}, {0x8643c0?, 0xc000014010?}, {0x8643e0?, 0xc000014018?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x77
github.com/docker/docker-credential-helpers/credentials.Serve({0x8678e8?, 0xc00003bc00?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xf3
main.main()
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:45 +0x194
panic: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc0000dd39d?, 0xfd1?})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x225
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x867e20?, 0xa9e6b8?}}, {0xc0000c2600, 0x2f})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:63 +0x28d
github.com/docker/docker-credential-helpers/credentials.Get({0x8678e8, 0xc000093bc0}, {0x8643c0?, 0xc0000a2000?}, {0x8643e0, 0xc0000a2008})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x20f
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x8678e8?, 0xc000093bc0?}, {0x7fff707a86ef?, 0xc0000cdef0?}, {0x8643c0?, 0xc0000a2000?}, {0x8643e0?, 0xc0000a2008?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x77
github.com/docker/docker-credential-helpers/credentials.Serve({0x8678e8?, 0xc000093bc0?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xf3
main.main()
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:45 +0x194
panic: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc00002b51d?, 0xfd1?})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x225
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x867e20?, 0xa9e6b8?}}, {0xc00002e8d0, 0x2f})
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:63 +0x28d
github.com/docker/docker-credential-helpers/credentials.Get({0x8678e8, 0xc00003bc00}, {0x8643c0?, 0xc000014010?}, {0x8643e0, 0xc000014018})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x20f
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x8678e8?, 0xc00003bc00?}, {0x7ffed37126ef?, 0xc0000afef0?}, {0x8643c0?, 0xc000014010?}, {0x8643e0?, 0xc000014018?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x77
github.com/docker/docker-credential-helpers/credentials.Serve({0x8678e8?, 0xc00003bc00?})
        github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xf3
main.main()
        github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:45 +0x194

The EKS pod identity setup is working properly.
If i run ecr docker login command, i can automatically access ECR and push and pull images. However, the whole purpose of the ecr credential helpers is to avoid to explicitly call the ecr docker login command.

[austinvazquez]

Hi @cwichka , for kubernetes I recommend the amazon-ecr-credential-provider. I'm not very familiar with EKS pod identity, but have you tried the provider?

See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/ for more details.

[Kern--]

Hi, just to clarify, you are trying to use this credential helper inside a pod with docker, not for pulling a pod on the node, right? The suggestion to use the amazon-ecr-credential-provider is for pulling the pod itself, not inside the pod. Sorry for the confusion.

What version of the credential helper were you using? It looks like v0.8.0 contains the change in the go SDK to to use EKS pod identity. Is anyone experiencing this issue using a version >= 0.8.0?