Follow semantic conventions for log records

[constanca-m]

Component(s)

extension/encoding/googlecloudlogentryencoding

Is your feature request related to a problem? Please describe.

Currently, the logs attribute names generated do not follow semantic conventions. The names are based on how the attributes are named in the payload.

We can check it by passing a simple log as an example:

{
  "httpRequest":{
    "cacheLookup":true,
    "latency":"0.464723s",
    "remoteIp":"122.231.191.3",
    "requestMethod":"GET",
    "requestSize":"225",
    "requestUrl":"https://34.110.216.199/",
    "responseSize":"262418",
    "status":200,
    "userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0"
  },
  "insertId":"1mv7h44f66w38p",
  "jsonPayload":{
    "@type":"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "backendTargetProjectNumber":"projects/8560181848",
    "cacheDecision":[
      "RESPONSE_HAS_CACHE_CONTROL",
      "RESPONSE_CACHE_CONTROL_DISALLOWED_CACHING",
      "RESPONSE_HAS_EXPIRES",
      "RESPONSE_HAS_CONTENT_TYPE",
      "CACHE_MODE_CACHE_ALL_STATIC"
    ],
    "cacheId":"LAX",
    "remoteIp":"122.231.191.3",
    "statusDetails":"response_sent_by_backend"
  },
  "logName":"projects/elastic-observability/logs/requests",
  "receiveTimestamp":"2025-06-29T13:59:06.899416987Z",
  "resource":{
    "labels":{
      "backend_service_name":"uninversal-profiling-symbols",
      "forwarding_rule_name":"with-tls",
      "project_id":"elastic-observability",
      "target_proxy_name":"profiling-cloudfront-symbols-target-proxy-2",
      "url_map_name":"profiling-cloudfront-symbols",
      "zone":"global"
    },
    "type":"http_load_balancer"
  },
  "severity":"INFO",
  "spanId":"2d84b769bfe8de6d",
  "timestamp":"2025-06-29T13:59:05.811699Z",
  "trace":"projects/elastic-observability/traces/6a4ff0a7376921a0eb18291d8618302c"
}

This generates:

resourceLogs:
  - resource:
      attributes:
        - key: gcp.resource_type
          value:
            stringValue: http_load_balancer
        - key: gcp.backend_service_name
          value:
            stringValue: uninversal-profiling-symbols
        - key: gcp.forwarding_rule_name
          value:
            stringValue: with-tls
        - key: gcp.project_id
          value:
            stringValue: elastic-observability
        - key: gcp.target_proxy_name
          value:
            stringValue: profiling-cloudfront-symbols-target-proxy-2
        - key: gcp.url_map_name
          value:
            stringValue: profiling-cloudfront-symbols
        - key: gcp.zone
          value:
            stringValue: global
    scopeLogs:
      - logRecords:
          - attributes:
              - key: gcp.http_request
                value:
                  kvlistValue:
                    values:
                      - key: cache_lookup
                        value:
                          boolValue: true
                      - key: latency
                        value:
                          stringValue: 0.464723s
                      - key: remote_ip
                        value:
                          stringValue: 122.231.191.3
                      - key: response_size
                        value:
                          intValue: "262418"
                      - key: status
                        value:
                          intValue: "200"
                      - key: user_agent
                        value:
                          stringValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0
                      - key: request_method
                        value:
                          stringValue: GET
                      - key: request_size
                        value:
                          intValue: "225"
                      - key: request_url
                        value:
                          stringValue: https://34.110.216.199/
              - key: log.record.uid
                value:
                  stringValue: 1mv7h44f66w38p
              - key: gcp.log_name
                value:
                  stringValue: projects/elastic-observability/logs/requests
            body:
              kvlistValue:
                values:
                  - key: '@type'
                    value:
                      stringValue: type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry
                  - key: backendTargetProjectNumber
                    value:
                      stringValue: projects/8560181848
                  - key: cacheDecision
                    value:
                      arrayValue:
                        values:
                          - stringValue: RESPONSE_HAS_CACHE_CONTROL
                          - stringValue: RESPONSE_CACHE_CONTROL_DISALLOWED_CACHING
                          - stringValue: RESPONSE_HAS_EXPIRES
                          - stringValue: RESPONSE_HAS_CONTENT_TYPE
                          - stringValue: CACHE_MODE_CACHE_ALL_STATIC
                  - key: cacheId
                    value:
                      stringValue: LAX
                  - key: remoteIp
                    value:
                      stringValue: 122.231.191.3
                  - key: statusDetails
                    value:
                      stringValue: response_sent_by_backend
            observedTimeUnixNano: "1751205546899416987"
            severityNumber: 9
            severityText: INFO
            spanId: 2d84b769bfe8de6d
            timeUnixNano: "1751205545811699000"
            traceId: 6a4ff0a7376921a0eb18291d8618302c
        scope: {}

Describe the solution you'd like

I would like the naming to follow semantic conventions.

Briefly, we can see names like:

• gcp.http_request.requestMethod -> rename to http.request.method
• gcp.http_request.requestSize -> rename to http.request.size
• gcp.zone -> cloud.region
• ...

I would like the encoding to handle all the common fields of the log entry and place them in attributes that follow the semantic conventions for fields names.

Describe alternatives you've considered

N/A

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• extension/encoding/googlecloudlogentryencoding: @alexvanboxel

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[constanca-m]

@alexvanboxel What do you think?

I have been reading the encoding extension in detail, and I would propose to do this in 3 steps:

1. We need to add more unit tests to make sure the step 2. works. This would be the first PR. I have a branch with this: main...constanca-m:opentelemetry-collector-contrib:gcpencoding-unittest-coverage. It uses the golden package to compare the expected with the return result, and it adds unit tests for every handle payload use case.

2. On the second PR, we start using a JSON struct instead of map[string] when unmarshaling the log. We will do this, because the fields are known. We need these known fields for step 3.

3. On the third PR, we start using the semantic conventions. This would be a breaking change.

[alexvanboxel]

I'm open to PR's the decoder. The purpose of spinning this out of Pub/Sub was to let it evolvo on it's own.

[alexvanboxel]

Note though, that when accepting, this should be defined as an option because attributes are not very well defined on log attributes... they have specific rules for spans, metrics, and "events"... but not arbitrary log lines. I'm fine accepting best effort mapping to semconv, but it should be configurable.