receiver/kafka: Improve performance and Kafka feature support

[marclop]

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

The current kafka receiver doesn't have the best performance, particularly when message marking is set, the performance is 4-5 times slower than when auto commit is enabled, looking at the implementation, this could be due to the interfaces exposed by the Sarama consumer. There are other Kafka Go clients that have increasingly more adoption and KIP support, like franz-go.

Additionally, the current Kafka receiver doesn't support consuming from a regex expression, forcing users with multi-tenant or multi-topic architectures to use the kafkatopicsobserver receiver creator, which creates 1 sarama Kafka client per topic (forcing 1 connection per client * number of brokers * topics, potentially more if there are more than 1 partition and client per topic).

Describe the solution you'd like

I'd like to propose that we consider opt-in support for franz-go, similar to #40364. Switching will improve performance and also add support for consuming from regex expressions, eliminating the need to use kafkatopicsobserver.

Describe alternatives you've considered

librdkafka uses cgo, which is undersireable for performance and allocations.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy @axw

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[marclop]

I'd like to work on the issue. My plan for implementing this is to split the work in two PRs. The first one will introduce generic processing that is applicable both to sarama / franz implementations and move the sarama receiver implementation to its own file.

The second PR will introduce an alpha featuregate with the franz-go implementation that behaves in the same way as the sarama receiver today.

[axw]

@marclop all yours :) Looking forward to seeing the benchmark results!