feat(http_client source): #23338 Enable AWS authentication for the http_client source. #23339

johannesfloriangeiger · 2025-07-04T14:38:33Z

Summary

See title, enables AWS authentication for the http_client source.

Vector configuration

[sources.http_client]
type = "http_client"
endpoint = "<ENDPOINT>"
auth = {strategy = "aws", auth = {}, service = "aps"}
method = "POST"
headers.Content-Type = ["application/x-www-form-urlencoded"]
query.query = "up"

[sinks.blackhole]
type = "blackhole"
inputs = ["http_client"]

How did you test this PR?

Manually. Get yourself an AWS Account, create an Amazon Managed Service for Prometheus workspace, get the Endpoint - query URL and use it in the config above. Run Vector and see it working (and not failing with a 403 like the current version).

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Closes: Enable AWS authentication for http_client source #23338

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

…rce.

src/aws/mod.rs

Copilot

Pull Request Overview

This PR adds AWS SigV4 signing support to the http_client source, allowing Vector to authenticate requests against AWS services (e.g., Amazon Managed Service for Prometheus).

Introduces conditional AWS signing in the http_client utility with empty-body support.
Refactors shared signing logic into create_signing_instructions and exposes sign_request_with_empty_body.
Updates HTTP sink config to streamline region and credentials provider setup and adds a changelog entry.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/sources/util/http_client.rs	Wraps request signing and send in timeouts, injects AWS SigV4 code
src/sinks/http/config.rs	Simplifies AWS region/credentials resolution in HTTP sink config
src/aws/mod.rs	Refactors signing into helper functions and adds empty-body signer
src/aws/auth.rs	Adds `region()` helper on `AwsAuthentication`
changelog.d/23338.feature.md	Adds changelog entry for AWS auth feature

Comments suppressed due to low confidence (1)

src/aws/auth.rs:372

The new region() helper on AwsAuthentication covers various enum variants. Adding unit tests for each variant would ensure region extraction remains correct under future changes.

    pub fn region(&self) -> Option<Region> {

src/sources/util/http_client.rs

src/aws/mod.rs

src/sources/util/http_client.rs

… 23338-http_client-aws-auth

pront · 2025-08-27T17:52:43Z

Please resolve the merge conflicts and we will take a look

johannesfloriangeiger · 2025-09-24T10:34:53Z

Please resolve the merge conflicts and we will take a look

Thanks, merged master back into my branch - the reorg of the imports has broken git, I try to keep changes to existing code to a minimum in a PR.

pront

Looks good. Found a few minor issues.

changelog.d/23338.feature.md

src/sinks/http/config.rs

src/aws/mod.rs

pront · 2025-09-29T20:59:58Z

src/sources/util/http_client.rs

    }
 }
+
+async fn prepare_request(auth: Option<Auth>, request: Request<Body>) -> Request<Body> {


With a little bit of refactoring this can be:

async fn prepare_request(auth: Option<Auth>, request: Request<Body>) -> Request<Body> { match auth { #[cfg(feature = "aws-core")] Some(Auth::Aws { auth, service: _ }) => sign_aws_request(auth, request).await, _ => request, } }

So, move the Some branch to a new function?

Co-authored-by: Pavlos Rontidis <[email protected]>

pront

Hi @johannesfloriangeiger, thanks again for this PR. Left some comments. The only real concern is that this is not tested anywhere in the codebase.

pront · 2025-11-05T18:05:21Z

src/sinks/http/config.rs

+                let region = auth
+                    .region()
+                    .or(default_region)
+                    .expect("Region must be specified");


Replace all expect() calls (excluding those in test code) with proper error propagation.

pront · 2025-11-05T18:18:33Z

src/sources/util/http_client.rs

+            let default_region = crate::aws::region_provider(&ProxyConfig::default(), None)
+                .expect("Region provider must be available")
+                .region()
+                .await;
+            let region = auth
+                .region()
+                .or(default_region)
+                .expect("Region must be specified");


Nit: introduce a helper and use it here and also while in config build()

pub async fn resolve_region_from_auth( auth: &AwsAuthentication, proxy_config: &ProxyConfig, tls_config: Option<&TlsConfig>, ) -> crate::Result<Region> { if let Some(region) = auth.region() { return Ok(region); } region_provider(proxy_config, tls_config)? .region() .await .ok_or_else(|| { "AWS region must be specified either in auth config or via environment/config".into() }) }

pront · 2025-11-05T18:19:08Z

src/sources/util/http_client.rs

    }
 }
+
+async fn prepare_request(auth: Option<Auth>, request: Request<Body>) -> Request<Body> {


src/aws/mod.rs

vectordotdev#23338: Enable AWS authentication for the http_client sou…

511bdbc

…rce.

johannesfloriangeiger requested a review from a team as a code owner July 4, 2025 14:38

github-actions bot added domain: sources Anything related to the Vector's sources domain: sinks Anything related to the Vector's sinks labels Jul 4, 2025

johannesfloriangeiger commented Jul 4, 2025

View reviewed changes

src/aws/mod.rs Show resolved Hide resolved

johannesfloriangeiger changed the title ~~feat(source: http_client): #23338 Enable AWS authentication for the http_client source.~~ feat(http_client source): #23338 Enable AWS authentication for the http_client source. Jul 4, 2025

pront requested a review from Copilot July 7, 2025 18:40

Copilot AI reviewed Jul 7, 2025

View reviewed changes

src/sources/util/http_client.rs Outdated Show resolved Hide resolved

src/sources/util/http_client.rs Outdated Show resolved Hide resolved

src/sources/util/http_client.rs Outdated Show resolved Hide resolved

src/aws/mod.rs Show resolved Hide resolved

pront reviewed Jul 7, 2025

View reviewed changes

src/aws/mod.rs Outdated Show resolved Hide resolved

src/sources/util/http_client.rs Outdated Show resolved Hide resolved

pront force-pushed the master branch 4 times, most recently from 1720078 to ffe54be Compare July 10, 2025 15:43

pront added the meta: awaiting author Pull requests that are awaiting their author. label Jul 10, 2025

vectordotdev#23338: Refactor code to improve readability.

3985c82

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Aug 9, 2025

johannesfloriangeiger added 4 commits August 10, 2025 12:49

Merge branch 'master' into 23338-http_client-aws-auth

03f3b99

Merge branch 'master' into 23338-http_client-aws-auth

16cfedf

vectordotdev#23338: Refactor code to improve readability.

712f2a0

Merge remote-tracking branch 'origin/23338-http_client-aws-auth' into…

ab10d38

… 23338-http_client-aws-auth

johannesfloriangeiger requested a review from pront August 16, 2025 17:20

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Aug 29, 2025

Merge branch 'master' into 23338-http_client-aws-auth

e6daa19

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Sep 24, 2025

johannesfloriangeiger added 4 commits September 25, 2025 11:30

Merge branch 'master' into 23338-http_client-aws-auth

049c35a

Merge branch 'master' into 23338-http_client-aws-auth

1a21be9

Merge branch 'master' into 23338-http_client-aws-auth

b6aa4c0

Merge branch 'master' into 23338-http_client-aws-auth

02fbd6f

Merge branch 'master' into 23338-http_client-aws-auth

5182b5d

pront approved these changes Sep 29, 2025

View reviewed changes

thomasqueirozb added source: http_client Anything `http_client` source related meta: awaiting author Pull requests that are awaiting their author. labels Sep 29, 2025

Merge branch 'master' into 23338-http_client-aws-auth

7ccc235

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Nov 1, 2025

johannesfloriangeiger and others added 5 commits November 1, 2025 17:18

Update changelog.d/23338.feature.md

1f665a4

Co-authored-by: Pavlos Rontidis <[email protected]>

Update src/aws/mod.rs

8590525

Co-authored-by: Pavlos Rontidis <[email protected]>

vectordotdev#23338: PR comments.

c5d2bd5

Merge branch 'master' into 23338-http_client-aws-auth

c59e173

Merge branch 'master' into 23338-http_client-aws-auth

20b230e

pront reviewed Nov 5, 2025

View reviewed changes

feat(http_client source): #23338 Enable AWS authentication for the http_client source. #23339

Are you sure you want to change the base?

feat(http_client source): #23338 Enable AWS authentication for the http_client source. #23339

Conversation

johannesfloriangeiger commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pront commented Aug 27, 2025

Uh oh!

johannesfloriangeiger commented Sep 24, 2025

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pront Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

johannesfloriangeiger Nov 1, 2025

Choose a reason for hiding this comment

Uh oh!

pront Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

pront Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

pront Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

pront Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johannesfloriangeiger commented Jul 4, 2025 •

edited

Loading