Skip to content

Commit a0d335d

Browse files
lahsivjarjmacd
andauthored
Allow num_consumers to be greater than 1 if batch enabled and partitioner defined (#13607)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description As per #12473, the number of consumers was set to `1` to prevent contention due to single-threaded batchers. However, with the introduction of [multiBatcher](https://github.com/open-telemetry/opentelemetry-collector/blob/aef8b1c3d26c062bc31e4f96c54f5f2a03b304c6/exporter/exporterhelper/internal/queuebatch/multi_batcher.go#L19) the batching will be done in multiple goroutines. Optimally a dynamic configuration to update the number of consumers to the number of partitions would be ideal, however, this could also cause contention as the queue is not partitioned and we can have hotspots where one partioner messages are more than other partitions. This PR makes it possible to configure the number of consumers to be greater than `1` statically (based on otel config) which should be better than what we have right now which bottlenecks the consumption even with partitioned batchers. <!-- Issue number if applicable --> #### Link to tracking issue Related to #12473 <!--Describe what testing was performed and which tests were added.--> #### Testing N/A <!--Describe the documentation added.--> #### Documentation N/A <!--Please delete paragraphs that you did not use before submitting.--> --------- Co-authored-by: Joshua MacDonald <[email protected]>
1 parent e6cedd0 commit a0d335d

File tree

2 files changed

+28
-3
lines changed

2 files changed

+28
-3
lines changed
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: enhancement
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. otlpreceiver)
7+
component: exporterhelper
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Respect `num_consumers` when batching and partitioning are enabled.
11+
12+
# One or more tracking issues or pull requests related to the change
13+
issues: [13607]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext:
19+
20+
# Optional: The change log or logs in which this entry should be included.
21+
# e.g. '[user]' or '[user, api]'
22+
# Include 'user' if the change is relevant to end users.
23+
# Include 'api' if there is a change to a library API.
24+
# Default: '[user]'
25+
change_logs: [user]

exporter/exporterhelper/internal/queuebatch/queue_batch.go

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,9 @@ func NewQueueBatch(
4949
if err != nil {
5050
return nil, err
5151
}
52-
if cfg.Batch.HasValue() {
53-
// If batching is enabled, keep the number of queue consumers to 1 if batching is enabled until we support
54-
// sharding as described in https://github.com/open-telemetry/opentelemetry-collector/issues/12473
52+
if cfg.Batch.HasValue() && set.Partitioner == nil {
53+
// If batching is enabled and partitioner is not defined then keep the number of queue consumers to 1.
54+
// see: https://github.com/open-telemetry/opentelemetry-collector/issues/12473
5555
cfg.NumConsumers = 1
5656
}
5757

0 commit comments

Comments
 (0)