aws_cloudwatch_logs to support `json/emf`

[hencrice]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

For now it's possible to send logs in EMF format to Amazon Cloudwatch using the aws_cloudwatch_logs sink. However, without the ability to configure the request header to x-amzn-logs-format: json/emf, the CloudWatch service can not correctly parse the EMF log. Can we expose this?

Attempted Solutions

sinks:
  emfLogSink:
    type: aws_cloudwatch_logs
    inputs:
      - emfLogSrc
    region: ${AWS_REGION}
    create_missing_group: false
    create_missing_stream: true
    group_name: ${EMF_LOG_GROUP_NAME}
    stream_name: ${TEST_RACK_NAME}
    buffer:
      type: memory 
      max_events: 500
      when_full: drop_newest
    batch:
      timeout_secs: 5
      max_events: 100
    encoding:
      codec: text

Proposal

I think having a boolean flag to turn on/off adding the x-amzn-logs-format: json/emf header would be sufficient.

References

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html#CloudWatch_Embedded_Metric_Format_Specification_PutLogEvents

Version

0.21.2

[jszwedko]

Thanks @hencrice ! I agree, this would be good to expose as an option. We have a precedent in the http sink of letting users set arbitrary headers (https://vector.dev/docs/reference/configuration/sinks/http/#request.headers). I could see allowing for the same here.

[hencrice]

Thanks for confirming. I browsed both the http sink and cloudwatch sinks to try to see where to make changes, probably because I'm too new to Rust, I'm not even sure how to start modifying https://github.com/vectordotdev/vector/blob/master/src/sinks/aws_cloudwatch_logs/config.rs#L98 since it does not have a header field. Then I'm not sure how to pass custom header to the AWS Rust SDK either.

[jszwedko]

Thanks for taking a peek!

For modifying the config to allow the option, I think you could reuse the same configuration struct the http sink uses here:

vector/src/sinks/http.rs

Line 91 in aa540a2

pub request: RequestConfig,

Which would replace the TowerRequestConfig. This is just a wrapper around TowerRequestConfig that adds the headers option:

vector/src/sinks/util/http.rs

Lines 521 to 529 in aa540a2

	/// A helper config struct
	#[derive(Deserialize, Serialize, Debug, Clone, Default)]
	#[serde(deny_unknown_fields)]
	pub struct RequestConfig {
	#[serde(flatten)]
	pub tower: TowerRequestConfig,
	#[serde(default)]
	pub headers: IndexMap<String, String>,
	}

For setting the header in the AWS SDK, you are right, I'm not seeing an easy way to do it. The Java SDK, for example, does have a builder method for overriding the request builder as shown on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_PutLogEvents.html . I'll open an issue on the upstream repository to get their thoughts.

[jszwedko]

I opened awslabs/aws-sdk-rust#537 to ask the SDK maintainers about overriding headers.

[jszwedko]

They responded there with a workaround for adding headers: awslabs/aws-sdk-rust#537 (comment)

[hencrice]

Thanks! I probably still need to do some digging to understand what she said. As owners of Vector, are you guys ok with the not-so-nice workaround now? Or you guys would prefer to wait until that's available? I might have some bandwidth to dig into this a few weeks later.

[jszwedko]

Thanks! I probably still need to do some digging to understand what she said. As owners of Vector, are you guys ok with the not-so-nice workaround now? Or you guys would prefer to wait until that's available? I might have some bandwidth to dig into this a few weeks later.

Thanks for your interest! I think the workaround is fine for now. We can switch to the fluent interface if/when they add it.

[hencrice]

@jszwedko I tried to follow the suggestion provided by the AWS Rust SDK team but got a bit stuck, could you help me figure out how to actually call into_parts from https://github.com/vectordotdev/vector/pull/12866/files#diff-04a3fbd5514b4e1f92e3a1148d001286eea9aff5479554f7e42921e4e3719bb2R224 ?

[jszwedko]

Hi @hencrice !

Thanks for taking a shot at it!

For calling into_parts I think we actually need to move away from using the fluent builder, as they mentioned:

Have users construct the operation directly (i.e. not using the fluent builder)

This would mean you want to create the input struct directly rather than using the builder: https://docs.rs/aws-sdk-cloudwatchlogs/latest/aws_sdk_cloudwatchlogs/input/struct.PutLogEventsInput.html#

Something like:

let input = PutLogEventsInput{
  log_group_name: Some(...)
  // initialize all fields
}

From there, you can call input.make_operation() to turn it into an Operation (https://docs.rs/aws-smithy-http/0.42.0/aws_smithy_http/operation/struct.Operation.html) which has into_request_response(). Once you have the Request part of that, you can call into_parts() on the request, modify the headers, call Request::from_parts(...) to turn it back into a Request and then Operation::from_parts() to take the modified request and turn that back into an Operation.

From there, you'll need to use the smithy client directly to run the operation and make the request. I'm still trying to track down how that works though.

[jszwedko]

Looks like to use the Operation you'll need to create a smithy client directly: https://docs.rs/aws-smithy-client/latest/aws_smithy_client/struct.Client.html (which has a call method).

We'll want to be careful to use the same configuration we use to create the CloudWatch Logs client though (region, retry policy, etc.).

Hopefully this helps point you in the right direction! Let me know if you have any other questions.