IAM role credentials not working if EC2 IMDSv2 session tokens 'required'

[fred-vogt]

EC2 Instance Metadata V2 support needs a newer version of github.com/aws/aws-sdk-go.

• https://github.com/aws/aws-sdk-go/releases/tag/v1.25.38
  • See "SDK Enhancements"
• docker updated to this version to fix the aws cloudwatch logs driver
  • moby/moby: issue #40422 / PR #40474

Tested with release 0.4.0.

~/.ecr/log/ecr-login.log:

time="2020-06-01T02:09:39Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=...
time="2020-06-01T02:09:39Z" level=error msg="Error retrieving credentials" 
error="
ecr: Failed to get authorization token: 
  NoCredentialProviders: no valid providers in chain. 
  Deprecated.
      For verbose messaging see aws.Config.CredentialsChainVerboseErrors
"

Making IMDSv2 session tokens optional fixes it:

AWS_PROFILE=admin-dev aws ec2 modify-instance-metadata-options \
    --instance-id <instance-id> \
    --http-tokens optional
{
    "InstanceId": "i-...",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "optional",
        "HttpPutResponseHopLimit": 64,
        "HttpEndpoint": "enabled"
    }
}

Notice the hop limit is NOT the default of 1.

[fred-vogt]

I tested this out with a custom build:

• https://github.com/fred-vogt/amazon-ecr-credential-helper/tree/IMDSv2/aws-sdk-go-1.31.7
  • https://github.com/fred-vogt/amazon-ecr-credential-helper/releases

NOTE: couldn't figure out how to update the existing dependency, so used go modules instead

The current latest github.com/aws/aws-sdk-go - 1.31.7 doesn't have this issue.

cat ~/.ecr/log/ecr-login.log 
time="2020-06-01T10:28:01Z" level=debug msg="Retrieving credentials" region=<region> registry=<acct> serverURL=<acct>.dkr.ecr.<region>.amazonaws.com
time="2020-06-01T10:28:01Z" level=debug msg="Checking file cache" registry=<acct>
time="2020-06-01T10:28:01Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=<acct>
time="2020-06-01T10:28:01Z" level=debug msg="Saving credentials to file cache" registry=<acct>

[samuelkarp]

Hi @fred-vogt, thanks for opening this issue! You're correct that the version of the AWS SDK needs to be upgraded in order to use IMDSv2. I've had this on my todo list for a while, but unfortunately haven't gotten around to it. My colleague opened #221 to move to Go modules, which should make it a bit easier to upgrade as dep isn't really popular anymore.

[samuelkarp]

I've merged #221 and the credential helper is now using Go modules. If you want to update the SDK and send a pull request, I'd be glad to have it. Note that we're currently still vendoring dependencies, so you'll need to run go mod vendor to sync the vendor directory.

[samuelkarp]

This should now be fixed in v0.5.0.

[kholisrag]

still not working tho

root@ip-10-11-77-69:~# echo "xxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com" | docker-credential-ecr-login get
credentials not found in native keychain