Consumer memory limit (queued.max.messages.kbytes)

[nikolaydimitrov]

Hello here!

For a high-level consumer, is there any dependency between queued.max.messages.kbytes and:

• Number of brokers?
• Number of partitions we are subscribed to?

Even with a setting of queued.max.messages.kbytes=1024 (1MB), I am still getting around 50MB of pulled messages until it stops pulling further. Which, reading the documentation, shouldn't be the case?!

For reference, I am subscribed to one topic of 800 partitions and we have 6 Kafka brokers.

[ireroba]

I had some success limiting max.partition.fetch.bytes. With 800 partitions, I'm betting it's grabbing a lot of data per partition initially.

[nikolaydimitrov]

Thanks for the tip