Support AWS ELB Access logs #41185

MichaelKatsoulis · 2025-07-08T15:54:49Z

Description

extension/encoding/awslogsencoding

The ELB(elastic load balancer) logs can have the following syntax based on their type:

This PR adds support for them so we can get valuable insights into traffic patterns, security, and application performance.

awslogs_encoding:
   format: elb_access_log

Describe alternatives you've considered

No response

Additional context

No response

Link to tracking issue

Fixes #40710

Testing

Documentation

As taken from README

ELB Access Log Fields

ELB access log record fields are mapped this way in the resulting OpenTelemetry log:

Application Load Balancer (ALB)

AWS Fields are according to documentation.

AWS Field	OpenTelemetry Field(s)
type	`network.protocol.name`
time	Log timestamp
elb	`cloud.resource_id`
client:port	`client.address`, `client.port`
received_bytes	`http.request.size`
sent_bytes	`http.response.size`
"request"	`url.full`, `http.request.method`, `network.protocol.version`
ssl_cipher	`tls.cipher`
ssl_protocol	`tls.protocol.version`
elb_status_code	`aws.elb.status.code`
target:port	Currently not supported
request_processing_time	Currently not supported
target_processing_time	Currently not supported
response_processing_time	Currently not supported
target_status_code	Currently not supported
"user_agent"	Currently not supported
target_group_arn	Currently not supported
"trace_id"	Currently not supported
"domain_name"	Currently not supported
"chosen_cert_arn"	Currently not supported
matched_rule_priority	Currently not supported
request_creation_time	Currently not supported
"actions_executed"	Currently not supported
"redirect_url"	Currently not supported
"error_reason"	Currently not supported
"target:port_list"	Currently not supported
"target_status_code_list"	Currently not supported
"classification"	Currently not supported
"classification_reason"	Currently not supported
conn_trace_id	Currently not supported

Network Load Balancer (NLB)

AWS Fields are according to documentation.

AWS Field	OpenTelemetry Field(s)
type	`network.protocol.name`
version	`network.protocol.version`
time	Log timestamp
elb	`cloud.resource_id`
listener	`aws.elb.tls.listener.resource_id`
client:port	`client.address`, `client.port`
received_bytes	`http.request.size`
sent_bytes	`http.response.size`
tls_cipher	`tls.cipher`
tls_protocol_version	`tls.protocol.version`
destination:port	Currently not supported
connection_time	Currently not supported
tls_handshake_time	Currently not supported
incoming_tls_alert	Currently not supported
chosen_cert_arn	Currently not supported
chosen_cert_serial	Currently not supported
tls_named_group	Currently not supported
domain_name	Currently not supported
alpn_fe_protocol	Currently not supported
alpn_be_protocol	Currently not supported
alpn_client_preference_list	Currently not supported
tls_connection_creation_time	Currently not supported

Classic Load Balancer (CLB)

AWS Fields are according to documentation

AWS Field	OpenTelemetry Field(s)
time	Log timestamp
elb	`cloud.resource_id`
client:port	`client.address`, `client.port`
elb_status_code	`aws.elb.status.code`
backend_status_code	`aws.elb.backend.status.code`
received_bytes	`http.request.size`
sent_bytes	`http.response.size`
"request"	`url.full`, `http.request.method`, `network.protocol.name`, `network.protocol.version`
ssl_cipher	`tls.cipher`
ssl_protocol	`tls.protocol.version`
backend:port	Currently not supported
request_processing_time	Currently not supported
backend_processing_time	Currently not supported
response_processing_time	Currently not supported
user_agent	Currently not supported

linux-foundation-easycla · 2025-07-08T15:54:57Z

The committers listed above are authorized under a signed CLA.

✅ login: MichaelKatsoulis / name: Michalis Katsoulis (ea1e31a, af144e8, 13e5bda, b70ce1c, 21402f8, b0d8f15, 462e554, 758c43b, eeee5ae, ddf1bd9, 3685de5, 1972e4d, 7cf4e13, cdf6ea5, d53c5f6, 63340cc, 70b6deb, f2adb1c, a1c9283, 763db4c)
✅ login: edmocosta / name: Edmo Vamerlatti Costa (932d281, cfb90b5)

constanca-m

Thanks Michalis! Good job. I took a very brief look, and noticed a couple of things.

constanca-m · 2025-07-08T16:35:16Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/unmarshaler.go

+
+	// Read first line to determine format
+	if !scanner.Scan() {
+		return plog.Logs{}, fmt.Errorf("no log lines found")


There is no test for this condition

constanca-m · 2025-07-08T16:35:31Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/unmarshaler.go

+	if err != nil {
+		return plog.Logs{}, fmt.Errorf("failed to parse log line: %w", err)
+	}


There is no test for this condition as well

constanca-m · 2025-07-08T16:35:47Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/unmarshaler.go

+	}
+
+	// Check for control message
+	if fields[0] == "Enable" {


Maybe make this Enable a constant?

constanca-m · 2025-07-08T16:38:19Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go

+	return value, nil
+}
+
+func findLogSyntaxByField(field string) (string, error) {


Can you add a comment on top of this function to explain this logic?

constanca-m · 2025-07-08T16:39:43Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go

+	if isValidTimestamp(field) {
+		return clbAccessLogs, nil
+	}


Here we are trying to run time.Parse for every log, time.Parse can be expensive. I would move this placement in the function to the default so that the other cheap conditions run first, like this:

func findLogSyntaxByField(field string) (string, error) { switch field { case "http", "https", "h2", "grpcs", "ws", "wss": return albAccessLogs, nil case "tls": return nlbAccessLogs, nil default: if isValidTimestamp(field) { return clbAccessLogs, nil } return "", fmt.Errorf("invalid type: %v", field) } }

Very good suggestion !

constanca-m · 2025-07-08T16:40:53Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/unmarshaler.go

+	epochNanoseconds, err := convertToUnixEpoch(albRecord.Time)
+	if err == nil {
+		recordLog.SetTimestamp(pcommon.Timestamp(epochNanoseconds))
+	} else {
+		f.logger.Debug("Timestamp cannot be converted to unix epoch nanoseconds", zap.Error(err))
+	}


If the timestamp is not valid, is there a reason to still record this log, or should the error just be thrown?

The same comment for the other 2 logs doing this as well.

I hadn't put much thought into this, but I think you are right. Because we are doing the same for the other fields in case they are not in the right format. For example in convertTextToNlbAccessLogRecord when a field processing fails we return an error that leads to no log record being created.

constanca-m · 2025-07-08T16:42:23Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go

+	}
+
+	// ELB status code and backend status code can be - in case of TCP and SSL entry
+	if fields[7] != "-" {


There is a constant for - so maybe use it here? And for all the others hardcoded "-"

constanca-m · 2025-07-08T16:47:54Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/unmarshaler.go

+	scanner := bufio.NewScanner(reader)
+
+	// Initialize scopeLogsByResource
+	scopeLogsByResource := map[string]plog.ScopeLogs{}


All logs coming from the same file should be part of the same resource. I don't think this map is needed

That is true! When I first created this I thought they could be mixed but the file is per load balancer. I will update the code.

...sion/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/benchmark_test.go

axw

Looks great. Please sign the CLA :)

extension/encoding/awslogsencodingextension/extension.go

axw · 2025-07-09T02:35:53Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/fields.go

+const (
+	AttributeELBStatusCode         = "aws.elb.status.code"         // int
+	AttributeELBBackendStatusCode  = "aws.elb.backend.status.code" // int
+	AttributeTlsListenerResourceID = "tls.listener.resource_id"    // string


Should this have an aws.elb. prefix? It seems fairly specific to AWS & ELB.

You are right, it is specific to nlb. I don't think this would be added in tls specific semantics conventions.

axw · 2025-07-09T02:40:40Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/unmarshaler.go

+	}
+	line = scanner.Text()
+
+	fields, err := extractFields(line)


Ideally we would iterate over the fields. We can optimise it later though.

axw · 2025-07-09T02:43:03Z

extension/encoding/awslogsencodingextension/README.md

+
+#### ELB Access Log Fields
+
+ELB access log record fields are mapped this way in the resulting OpenTelemetry log:


There are more fields in ELB access logs that are not mentioned below. Seems we're dropping them at the moment. Can you please add a list of the fields that are not translated, and mention that they're TODO?

I updated the documentation!

Co-authored-by: Andrew Wilkins <[email protected]>

…er/elb-access-log/benchmark_test.go Co-authored-by: Constança Manteigas <[email protected]>

…emetry-collector-contrib into elb-access-logs

MichaelKatsoulis · 2025-07-09T13:08:11Z

Looks great. Please sign the CLA :)

I am waiting for the authorization!

atoulme · 2025-07-16T22:37:58Z

Please take a look at the CI failures

constanca-m · 2025-07-18T13:45:46Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/fields.go

+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package elbaccesslogs


This file has no unit tests. Can you add some?

This file only defines three constants. Are tests for these really useful? I don't see tests for the s3-access-logs fields.go.

I could add some though, no problem.

MichaelKatsoulis · 2025-07-22T12:23:09Z

Hi @atoulme , could you please approve the workflow another time to check if there are remaining CI failures?

…emetry-collector-contrib into elb-access-logs

MichaelKatsoulis · 2025-07-29T08:46:53Z

@atoulme or @andrzej-stencel since I already have an approval, could you give an approval for the CI to run and if it succeeds we can merge this one?

MichaelKatsoulis · 2025-07-29T14:20:40Z

@andrzej-stencel could I have another workflow approval. I think I got rid of the last linter issue.

MichaelKatsoulis · 2025-07-30T08:54:37Z

@evan-bradley whenever possible, could I get a final review and approval so we can merge this one?

dehaansa · 2025-07-30T18:43:43Z

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go

+// forward until it finds the ending quote, and considers
+// that just 1 value. Otherwise, it returns the value as
+// it is.
+func scanField(logLine string) (string, string, error) {


Have you considered replacing this complex scanning logic with a csv reader, using a space as the delimiter?

This scan field logic is used to optimize performance, csv reader wouldn't be good for our use case (there is a benchmark test to check this)

constanca-m · 2025-08-04T14:40:16Z

@atoulme @edmocosta @dehaansa Please, do you have time to merge this PR? Thanks

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go

…n-telemetry#41185)  #### Description extension/encoding/awslogsencoding The ELB(elastic load balancer) logs can have the following syntax based on their type: - [Network Load Balancer Access Logs](https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html#access-log-entry-format) - [Classic Load Balancer Access Logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/access-log-collection.html#access-log-entry-format) - [Application Load Balancer Connection Logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-connection-logs.html#connection-log-entry-format) This PR adds support for them so we can get valuable insights into traffic patterns, security, and application performance. ``` awslogs_encoding: format: elb_access_log ``` ### Describe alternatives you've considered _No response_ ### Additional context _No response_  #### Link to tracking issue Fixes open-telemetry#40710  #### Testing  #### Documentation As taken from README #### ELB Access Log Fields ELB access log record fields are mapped this way in the resulting OpenTelemetry log: ##### Application Load Balancer (ALB) > AWS Fields are according to [documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html). | **AWS Field** | **OpenTelemetry Field(s)** | |------------------------------|-----------------------------------------------------------------------| | type | `network.protocol.name` | | time | Log timestamp | | elb | `cloud.resource_id` | | client:port | `client.address`, `client.port` | | received_bytes | `http.request.size` | | sent_bytes | `http.response.size` | | "request" | `url.full`, `http.request.method`, `network.protocol.version` | | ssl_cipher | `tls.cipher` | | ssl_protocol | `tls.protocol.version` | | elb_status_code | `aws.elb.status.code` | | target:port | _Currently not supported_ | | request_processing_time | _Currently not supported_ | | target_processing_time | _Currently not supported_ | | response_processing_time | _Currently not supported_ | | target_status_code | _Currently not supported_ | | "user_agent" | _Currently not supported_ | | target_group_arn | _Currently not supported_ | | "trace_id" | _Currently not supported_ | | "domain_name" | _Currently not supported_ | | "chosen_cert_arn" | _Currently not supported_ | | matched_rule_priority | _Currently not supported_ | | request_creation_time | _Currently not supported_ | | "actions_executed" | _Currently not supported_ | | "redirect_url" | _Currently not supported_ | | "error_reason" | _Currently not supported_ | | "target:port_list" | _Currently not supported_ | | "target_status_code_list" | _Currently not supported_ | | "classification" | _Currently not supported_ | | "classification_reason" | _Currently not supported_ | | conn_trace_id | _Currently not supported_ | ##### Network Load Balancer (NLB) > AWS Fields are according to [documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest//network/load-balancer-access-logs.html#access-log-entry-format). | **AWS Field** | **OpenTelemetry Field(s)** | |------------------------------|-------------------------------------------------------------| | type | `network.protocol.name` | | version | `network.protocol.version` | | time | Log timestamp | | elb | `cloud.resource_id` | | listener | `aws.elb.tls.listener.resource_id` | | client:port | `client.address`, `client.port` | | received_bytes | `http.request.size` | | sent_bytes | `http.response.size` | | tls_cipher | `tls.cipher` | | tls_protocol_version | `tls.protocol.version` | | destination:port | _Currently not supported_ | | connection_time | _Currently not supported_ | | tls_handshake_time | _Currently not supported_ | | incoming_tls_alert | _Currently not supported_ | | chosen_cert_arn | _Currently not supported_ | | chosen_cert_serial | _Currently not supported_ | | tls_named_group | _Currently not supported_ | | domain_name | _Currently not supported_ | | alpn_fe_protocol | _Currently not supported_ | | alpn_be_protocol | _Currently not supported_ | | alpn_client_preference_list | _Currently not supported_ | | tls_connection_creation_time | _Currently not supported_ | ##### Classic Load Balancer (CLB) > AWS Fields are according to [documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/access-log-collection.html) | **AWS Field** | **OpenTelemetry Field(s)** | |-----------------------|--------------------------------------------------------------------------------------------| | time | Log timestamp | | elb | `cloud.resource_id` | | client:port | `client.address`, `client.port` | | elb_status_code | `aws.elb.status.code` | | backend_status_code | `aws.elb.backend.status.code` | | received_bytes | `http.request.size` | | sent_bytes | `http.response.size` | | "request" | `url.full`, `http.request.method`, `network.protocol.name`, `network.protocol.version` | | ssl_cipher | `tls.cipher` | | ssl_protocol | `tls.protocol.version` | | backend:port | _Currently not supported_ | | request_processing_time | _Currently not supported_ | | backend_processing_time | _Currently not supported_ | | response_processing_time | _Currently not supported_ | | user_agent | _Currently not supported_ |  --------- Co-authored-by: Andrew Wilkins <[email protected]> Co-authored-by: Constança Manteigas <[email protected]> Co-authored-by: Edmo Vamerlatti Costa <[email protected]>

MichaelKatsoulis added 3 commits July 8, 2025 18:46

elb-access-log initial commit

af144e8

Update tests

13e5bda

make fmt

462e554

MichaelKatsoulis requested a review from a team as a code owner July 8, 2025 15:54

MichaelKatsoulis requested a review from evan-bradley July 8, 2025 15:54

MichaelKatsoulis marked this pull request as draft July 8, 2025 15:54

github-actions bot assigned dehaansa Jul 8, 2025

github-actions bot added the extension/encoding/awslogsencoding label Jul 8, 2025

github-actions bot requested review from axw and constanca-m July 8, 2025 15:55

constanca-m reviewed Jul 8, 2025

View reviewed changes

...sion/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/benchmark_test.go Outdated Show resolved Hide resolved

axw reviewed Jul 9, 2025

View reviewed changes

MichaelKatsoulis and others added 6 commits July 9, 2025 11:20

Update extension/encoding/awslogsencodingextension/extension.go

b0d8f15

Co-authored-by: Andrew Wilkins <[email protected]>

Review updates

b70ce1c

Update extension/encoding/awslogsencodingextension/internal/unmarshal…

21402f8

…er/elb-access-log/benchmark_test.go Co-authored-by: Constança Manteigas <[email protected]>

Merge branch 'elb-access-logs' of github.com:MichaelKatsoulis/opentel…

ea1e31a

…emetry-collector-contrib into elb-access-logs

Update documentation of fields

f2adb1c

Merge branch 'main' into elb-access-logs

3685de5

Add missing tests

70b6deb

MichaelKatsoulis marked this pull request as ready for review July 9, 2025 13:29

github-actions bot assigned edmocosta Jul 9, 2025

MichaelKatsoulis requested review from axw and constanca-m July 9, 2025 13:29

Merge branch 'main' into elb-access-logs

a1c9283

atoulme added the waiting-for-code-owners label Jul 16, 2025

constanca-m reviewed Jul 18, 2025

View reviewed changes

MichaelKatsoulis added 2 commits July 21, 2025 12:32

Merge remote-tracking branch 'upstream/main' into elb-access-logs

eeee5ae

Run make goporto

cdf6ea5

MichaelKatsoulis added 2 commits July 24, 2025 16:34

Fix compiler error and add changelog

763db4c

Merge branch 'main' into elb-access-logs

758c43b

axw approved these changes Jul 25, 2025

View reviewed changes

MichaelKatsoulis added 2 commits July 28, 2025 11:05

Fix linter errors

7cf4e13

Merge branch 'elb-access-logs' of github.com:MichaelKatsoulis/opentel…

ddf1bd9

…emetry-collector-contrib into elb-access-logs

Get rid of variable shadowing issue

1972e4d

dehaansa reviewed Jul 30, 2025

View reviewed changes

Merge branch 'main' into elb-access-logs

63340cc

dehaansa approved these changes Aug 4, 2025

View reviewed changes

dehaansa added ready to merge Code review completed; ready to merge by maintainers and removed waiting-for-code-owners labels Aug 4, 2025

Merge branch 'main' into elb-access-logs

cfb90b5

edmocosta reviewed Aug 5, 2025

View reviewed changes

extension/encoding/awslogsencodingextension/internal/unmarshaler/elb-access-log/elb.go Outdated Show resolved Hide resolved

MichaelKatsoulis and others added 2 commits August 5, 2025 14:39

make nlb, clb and alb acronyms uppercase

d53c5f6

Merge branch 'main' into elb-access-logs

932d281

edmocosta merged commit aa81338 into open-telemetry:main Aug 5, 2025
181 checks passed


		#### ELB Access Log Fields

		ELB access log record fields are mapped this way in the resulting OpenTelemetry log:

Support AWS ELB Access logs #41185

Support AWS ELB Access logs #41185

Uh oh!

Conversation

MichaelKatsoulis commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Describe alternatives you've considered

Additional context

Link to tracking issue

Testing

Documentation

ELB Access Log Fields

Application Load Balancer (ALB)

Network Load Balancer (NLB)

Classic Load Balancer (CLB)

Uh oh!

linux-foundation-easycla bot commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

constanca-m left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

constanca-m Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

axw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelKatsoulis commented Jul 9, 2025

Uh oh!

atoulme commented Jul 16, 2025

Uh oh!

constanca-m Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelKatsoulis commented Jul 22, 2025

Uh oh!

MichaelKatsoulis commented Jul 29, 2025

Uh oh!

MichaelKatsoulis commented Jul 29, 2025

Uh oh!

MichaelKatsoulis commented Jul 30, 2025

Uh oh!

MichaelKatsoulis commented Jul 8, 2025 •

edited

Loading

linux-foundation-easycla bot commented Jul 8, 2025 •

edited

Loading

constanca-m Jul 8, 2025 •

edited

Loading

constanca-m Jul 18, 2025 •

edited

Loading

constanca-m commented Aug 4, 2025 •

edited

Loading