Skip to content

Commit e7c3295

Browse files
authored
[GCP encoding] Use JSON struct for all common fields and improve performance (#41467)
#### Description This PR uses a JSON struct now instead of `map[string]any`, as we know what to expect from a GCP log entry. This causes significant performance improvement: ``` goos: linux goarch: amd64 pkg: github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/googlecloudlogentryencodingextension cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ Test-16 19.907µ ± 3% 6.583µ ± 4% -66.93% (p=0.000 n=10) │ old.txt │ new.txt │ │ B/op │ B/op vs base │ Test-16 13.310Ki ± 0% 5.326Ki ± 0% -59.98% (p=0.000 n=10) │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ Test-16 340.00 ± 0% 90.00 ± 0% -73.53% (p=0.000 n=10) ``` The main goal of using a JSON struct is so that we can use semantic conventions, like mentioned in issue #41087. There are also two breaking changes, since the code implemented before was not working correctly for these cases: - Logs also have now observedTimeUnixNano and flags - If there is an error handling the log, just return the error and empty logs for ALL cases. Before this behavior was inconsistent: either the error was ignored ([here](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/5943f5a933843e6f81bed28a05e2f33d8c1fc485/extension/encoding/googlecloudlogentryencodingextension/extension.go#L86)), or it would stop the logs from being received ([here](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/5943f5a933843e6f81bed28a05e2f33d8c1fc485/extension/encoding/googlecloudlogentryencodingextension/extension.go#L94)) <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Relates to #41087 (comment). <!--Describe what testing was performed and which tests were added.--> #### Testing Unit tests fixed. <!--Describe the documentation added.--> #### Documentation N/A.
1 parent 7ee72d1 commit e7c3295

File tree

13 files changed

+438
-443
lines changed

13 files changed

+438
-443
lines changed

.chloggen/gcp-add-common-fields.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: breaking
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
7+
component: googlecloudlogentry_encoding
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Add 'observedTimestamp' and 'flags' field to resulting log, and throw error if log failed to parse.
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [41467]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext:
19+
20+
# If your change doesn't affect end users or the exported elements of any package,
21+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
22+
# Optional: The change log or logs in which this entry should be included.
23+
# e.g. '[user]' or '[user, api]'
24+
# Include 'user' if the change is relevant to end users.
25+
# Include 'api' if there is a change to a library API.
26+
# Default: '[user]'
27+
change_logs: []
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
// Copyright The OpenTelemetry Authors
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
package googlecloudlogentryencodingextension
5+
6+
import (
7+
"bytes"
8+
"os"
9+
"testing"
10+
11+
gojson "github.com/goccy/go-json"
12+
"github.com/stretchr/testify/require"
13+
)
14+
15+
func BenchmarkTest(b *testing.B) {
16+
name := "testdata/real_log.json"
17+
data, err := os.ReadFile(name)
18+
require.NoError(b, err)
19+
20+
var dest bytes.Buffer
21+
err = gojson.Compact(&dest, data)
22+
require.NoError(b, err)
23+
24+
ex := &ext{}
25+
26+
b.ResetTimer()
27+
b.ReportAllocs()
28+
for i := 0; i < b.N; i++ {
29+
_, err := ex.UnmarshalLogs(dest.Bytes())
30+
require.NoError(b, err)
31+
}
32+
}

extension/encoding/googlecloudlogentryencodingextension/extension.go

Lines changed: 16 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,9 @@ package googlecloudlogentryencodingextension // import "github.com/open-telemetr
55

66
import (
77
"context"
8-
stdjson "encoding/json"
8+
"fmt"
99

10+
gojson "github.com/goccy/go-json"
1011
"go.opentelemetry.io/collector/component"
1112
"go.opentelemetry.io/collector/pdata/plog"
1213

@@ -16,31 +17,14 @@ import (
1617
var _ encoding.LogsUnmarshalerExtension = (*ext)(nil)
1718

1819
type ext struct {
19-
Config Config
20-
extractFns map[string]extractFn
20+
config Config
2121
}
2222

2323
func newExtension(cfg *Config) *ext {
24-
return &ext{Config: *cfg}
24+
return &ext{config: *cfg}
2525
}
2626

27-
func (ex *ext) Start(_ context.Context, _ component.Host) error {
28-
ex.extractFns = map[string]extractFn{
29-
"protoPayload": handleProtoPayload,
30-
"receiveTimestamp": handleReceiveTimestamp,
31-
"timestamp": handleTimestamp,
32-
"insertId": handleInsertID,
33-
"logName": handleLogName,
34-
"textPayload": handleTextPayload,
35-
"jsonPayload": handleJSONPayload,
36-
"severity": handleSeverity,
37-
"trace": handleTrace,
38-
"spanId": handleSpanID,
39-
"traceSampled": handleTraceSampled,
40-
"labels": handleLabels,
41-
"httpRequest": handleHTTPRequest,
42-
"resource": handleResource,
43-
}
27+
func (*ext) Start(_ context.Context, _ component.Host) error {
4428
return nil
4529
}
4630

@@ -57,40 +41,26 @@ func (ex *ext) UnmarshalLogs(buf []byte) (plog.Logs, error) {
5741
return logs, nil
5842
}
5943

60-
// TranslateLogEntry translates a JSON-encoded LogEntry message into a pair of
61-
// pcommon.Resource and plog.LogRecord, trying to keep as close as possible to
62-
// the semantic conventions.
44+
// TranslateLogEntry translates a JSON-encoded LogEntry message into plog.Logs,
45+
// trying to keep as close as possible to the GCP original names.
6346
//
6447
// For maximum fidelity, the decoding is done according to the protobuf message
6548
// schema; this ensures that a numeric value in the input is correctly
6649
// translated to either an integer or a double in the output. It falls back to
6750
// plain JSON decoding if payload type is not available in the proto registry.
6851
func (ex *ext) translateLogEntry(data []byte) (plog.Logs, error) {
69-
var src map[string]stdjson.RawMessage
70-
err := json.Unmarshal(data, &src)
71-
if err != nil {
72-
return plog.Logs{}, err
52+
var log logEntry
53+
if err := gojson.Unmarshal(data, &log); err != nil {
54+
return plog.Logs{}, fmt.Errorf("failed to unmarshal log entry: %w", err)
7355
}
7456

7557
logs := plog.NewLogs()
76-
rl := logs.ResourceLogs().AppendEmpty()
77-
res := rl.Resource()
78-
lr := rl.ScopeLogs().AppendEmpty().LogRecords().AppendEmpty()
58+
resourceLogs := logs.ResourceLogs().AppendEmpty()
59+
resource := resourceLogs.Resource()
60+
logRecord := resourceLogs.ScopeLogs().AppendEmpty().LogRecords().AppendEmpty()
7961

80-
resAttrs := res.Attributes()
81-
attrs := lr.Attributes()
82-
83-
for key, value := range src {
84-
fn := ex.extractFns[key]
85-
if fn != nil {
86-
kverr := fn(resAttrs, lr, attrs, key, value, ex.Config)
87-
if kverr == nil {
88-
delete(src, key)
89-
}
90-
}
62+
if err := handleLogEntryFields(resource.Attributes(), logRecord, log, ex.config); err != nil {
63+
return plog.Logs{}, fmt.Errorf("failed to handle log entry: %w", err)
9164
}
92-
93-
// All other fields -> Attributes["gcp.*"]
94-
err = translateInto(attrs, getLogEntryDescriptor(), src, preserveDst, prefixKeys("gcp."), snakeifyKeys)
95-
return logs, err
65+
return logs, nil
9666
}

extension/encoding/googlecloudlogentryencodingextension/extension_test.go

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,7 @@ func TestUnmarshalLogs(t *testing.T) {
5656
data := readFile(t, "testdata/real_log.json")
5757
return bytes.Replace(data, []byte(`"spanId":"3e3a5741b18f0710"`), []byte(`"spanId":"13210305202245662348"`), 1)
5858
}(),
59-
wantFile: "testdata/invalid_span_expected.yaml",
60-
setFlag: true,
61-
cfg: *createDefaultConfig().(*Config),
59+
expectsErr: "failed to handle log entry",
6260
},
6361
}
6462

extension/encoding/googlecloudlogentryencodingextension/go.mod

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,8 @@ module github.com/open-telemetry/opentelemetry-collector-contrib/extension/encod
33
go 1.23.0
44

55
require (
6-
cloud.google.com/go/logging v1.13.0
6+
github.com/goccy/go-json v0.10.5
77
github.com/iancoleman/strcase v0.3.0
8-
github.com/json-iterator/go v1.1.12
98
github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding v0.131.0
109
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden v0.131.0
1110
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest v0.131.0
@@ -19,13 +18,11 @@ require (
1918
go.opentelemetry.io/collector/pdata v1.37.0
2019
go.uber.org/goleak v1.3.0
2120
google.golang.org/genproto v0.0.0-20250715232539-7130f93afb79
22-
google.golang.org/genproto/googleapis/api v0.0.0-20250715232539-7130f93afb79
2321
google.golang.org/protobuf v1.36.6
2422
)
2523

2624
require (
2725
cloud.google.com/go/iam v1.5.2 // indirect
28-
cloud.google.com/go/longrunning v0.6.7 // indirect
2926
github.com/cespare/xxhash/v2 v2.3.0 // indirect
3027
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
3128
github.com/go-logr/logr v1.4.3 // indirect
@@ -35,6 +32,7 @@ require (
3532
github.com/gogo/protobuf v1.3.2 // indirect
3633
github.com/google/uuid v1.6.0 // indirect
3734
github.com/hashicorp/go-version v1.7.0 // indirect
35+
github.com/json-iterator/go v1.1.12 // indirect
3836
github.com/knadh/koanf/maps v0.1.2 // indirect
3937
github.com/knadh/koanf/providers/confmap v1.0.0 // indirect
4038
github.com/knadh/koanf/v2 v2.2.2 // indirect
@@ -61,6 +59,7 @@ require (
6159
golang.org/x/net v0.41.0 // indirect
6260
golang.org/x/sys v0.33.0 // indirect
6361
golang.org/x/text v0.26.0 // indirect
62+
google.golang.org/genproto/googleapis/api v0.0.0-20250715232539-7130f93afb79 // indirect
6463
google.golang.org/genproto/googleapis/rpc v0.0.0-20250707201910-8d1bb00bc6a7 // indirect
6564
google.golang.org/grpc v1.74.2 // indirect
6665
gopkg.in/yaml.v3 v3.0.1 // indirect

extension/encoding/googlecloudlogentryencodingextension/go.sum

Lines changed: 2 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)