Default credential loader doesn't seem to follow 301s from IMDSv2

[jszwedko]

Describe the bug

This is an indirect bug report from a user of Vector, which uses the AWS SDK.

When using the default credential chain, they see the loader reporting that it saw a 301, but it doesn't seem to automatically follow it, as I would expect.

The user is using KIAM to intercept the requests which is likely why the behavior differs from IMDSv2 generally.

See debug logs below.

Expected Behavior

The default credentials loader follows any observed 301s to fetch the IAM credentials.

Current Behavior

When a 301 is observed, it stops and bubbles up an error.

Reproduction Steps

Unfortunately I haven't observed this behavior myself, but we could ask the user for additional details if necessary.

Possible Solution

Default credential loader follows 301s from IMDSv2 API.

Additional Information/Context

User is running KIAM to intercept the metadata requests.

Vector issue: vectordotdev/vector#13211

Version

├── aws-config v0.11.0
│   ├── aws-http v0.11.0
│   │   ├── aws-smithy-http v0.41.0
│   │   │   ├── aws-smithy-eventstream v0.41.0
│   │   │   │   ├── aws-smithy-types v0.41.0
│   │   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-types v0.11.0
│   │   │   ├── aws-smithy-async v0.41.0
│   │   │   ├── aws-smithy-client v0.41.0
│   │   │   │   ├── aws-smithy-async v0.41.0 (*)
│   │   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.41.0
│   │   │   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   │   ├── aws-smithy-types v0.41.0 (*)
│   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-sdk-sso v0.11.0
│   │   ├── aws-endpoint v0.11.0
│   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-types v0.11.0 (*)
│   │   ├── aws-http v0.11.0 (*)
│   │   ├── aws-sig-auth v0.11.0
│   │   │   ├── aws-sigv4 v0.11.0
│   │   │   │   ├── aws-smithy-eventstream v0.41.0 (*)
│   │   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.41.0 (*)
│   │   │   ├── aws-smithy-http v0.41.0 (*)
│   │   │   ├── aws-types v0.11.0 (*)
│   │   ├── aws-smithy-async v0.41.0 (*)
│   │   ├── aws-smithy-client v0.41.0 (*)
│   │   ├── aws-smithy-http v0.41.0 (*)
│   │   ├── aws-smithy-http-tower v0.41.0 (*)
│   │   ├── aws-smithy-json v0.41.0
│   │   │   └── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-types v0.11.0 (*)
│   ├── aws-sdk-sts v0.11.0
│   │   ├── aws-endpoint v0.11.0 (*)
│   │   ├── aws-http v0.11.0 (*)
│   │   ├── aws-sig-auth v0.11.0 (*)
│   │   ├── aws-smithy-async v0.41.0 (*)
│   │   ├── aws-smithy-client v0.41.0 (*)
│   │   ├── aws-smithy-http v0.41.0 (*)
│   │   ├── aws-smithy-http-tower v0.41.0 (*)
│   │   ├── aws-smithy-query v0.41.0
│   │   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-types v0.41.0 (*)
│   │   ├── aws-smithy-xml v0.41.0
│   │   ├── aws-types v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-cloudwatch v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-query v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-smithy-xml v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-cloudwatchlogs v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-firehose v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-kinesis v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-json v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-s3 v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-sigv4 v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-eventstream v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-smithy-xml v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sdk-sqs v0.11.0
│   ├── aws-endpoint v0.11.0 (*)
│   ├── aws-http v0.11.0 (*)
│   ├── aws-sig-auth v0.11.0 (*)
│   ├── aws-smithy-async v0.41.0 (*)
│   ├── aws-smithy-client v0.41.0 (*)
│   ├── aws-smithy-http v0.41.0 (*)
│   ├── aws-smithy-http-tower v0.41.0 (*)
│   ├── aws-smithy-query v0.41.0 (*)
│   ├── aws-smithy-types v0.41.0 (*)
│   ├── aws-smithy-xml v0.41.0 (*)
│   ├── aws-types v0.11.0 (*)
├── aws-sigv4 v0.11.0 (*)
├── aws-smithy-async v0.41.0 (*)
├── aws-smithy-client v0.41.0 (*)
├── aws-smithy-http v0.41.0 (*)
├── aws-smithy-http-tower v0.41.0 (*)
├── aws-smithy-types v0.41.0 (*)
├── aws-types v0.11.0 (*)

Environment details (OS name and version, etc.)

x86_64-unknown-linux-musl

Logs

2022-06-21T15:09:44.896310Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=environment variable not set
2022-06-21T15:09:44.896352Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2022-06-21T15:09:44.896368Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-06-21T15:09:44.896379Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-06-21T15:09:44.896429Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-06-21T15:09:44.896447Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-06-21T15:09:44.896458Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-06-21T15:09:44.896477Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-06-21T15:09:44.896487Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-06-21T15:09:44.896493Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-06-21T15:09:44.896511Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=No profiles were defined
2022-06-21T15:09:44.896529Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=$AWS_WEB_IDENTITY_TOKEN_FILE was not set
2022-06-21T15:09:44.896543Z  INFO provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=ECS provider not configured
2022-06-21T15:09:44.896558Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2022-06-21T15:09:44.896567Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::fs_util: loaded home directory src="HOME"
2022-06-21T15:09:44.896576Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-06-21T15:09:44.896585Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-06-21T15:09:44.896595Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-06-21T15:09:44.896609Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-06-21T15:09:44.896619Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-06-21T15:09:44.896628Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-06-21T15:09:44.896644Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-06-21T15:09:44.896650Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-06-21T15:09:44.898362Z  WARN provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider failed to provide credentials provider=Ec2InstanceMetadata error=An error occurred while loading credentials: Error response from IMDS (code: 301). Response { status: 301, version: HTTP/1.1, headers: {"content-type": "text/html; charset=utf-8", "location": "/latest/meta-data/iam/security-credentials/", "date": "Tue, 21 Jun 2022 15:09:44 GMT", "content-length": "78"}, body: SdkBody { inner: Once(Some(b"<a href=\"/latest/meta-data/iam/security-credentials/\">Moved Permanently</a>.\n\n")), retryable: true } }

[rcoh]

generally we don't follow 301s—I'll need to consult the spec and figure out if this behavior should be honored or not. Thanks for reporting, will follow up soon

[rcoh]

I followed up—I suspect the issue is not the 301—Other SDKs don't follow 301s for IMDS (and if they do it's a bug / security issue).

The issue instead seems to be that we're hitting security-credentials (no trailing slash) and KIAM is (for some reason) deciding to redirect to security-credentials/.

I don't see any particular reason we couldn't hit the URL with a trailing slash from the IMDS client though

[jszwedko]

Thanks for the response! I can see why following 301s could be considered a security concern.

uswitch/kiam#172 seems relevant. They switched to returning a 301 from a 308 thinking that clients would handle it better.

uswitch/kiam#116 also seems relevant.

I spun up an EC2 instance to try it out and both with and without / seem to return 200s at the current time for IMDSv2 (I didn't try KIAM):

[ec2-user@ip-172-31-43-67 ~]$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v 169.254.169.254/latest/meta-data/iam/security-credentials
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 23
< Content-Type: text/plain
< Date: Wed, 22 Jun 2022 18:32:15 GMT
< Last-Modified: Wed, 22 Jun 2022 18:31:24 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0
cloudtrail-ec2-instance

[ec2-user@ip-172-31-43-67 ~]$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v 169.254.169.254/latest/meta-data/iam/security-credentials/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 23
< Content-Type: text/plain
< Date: Wed, 22 Jun 2022 18:32:17 GMT
< Last-Modified: Wed, 22 Jun 2022 18:31:24 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0
cloudtrail-ec2-instance

curling the parent path,/iam, seems to return the "directory entries" with the trailing /:

$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v 169.254.169.254/latest/meta-data/iam/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Content-Length: 26
< Content-Type: text/plain
< Date: Wed, 22 Jun 2022 18:34:39 GMT
< Last-Modified: Wed, 22 Jun 2022 18:31:24 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0
info
security-credentials/

I'll leave it to you and others with more intimate knowledge of IMDSv2 but it does seem like having the client hit the endpoint with a trailing / would be appropriate.

[rcoh]

it seems like KIAM explicitly added this redirect for some internal reason: uswitch/kiam#121

[rcoh]

PR: smithy-lang/smithy-rs#1487
Unfortunately it's not going to make the release that's going out today but will go out in the next one

[jszwedko]

Awesome, thanks for the quick review and quick fix @rcoh !

[rcoh]

this shipped!