Skip to content

Conversation

@shamanu4
Copy link

Description

This PR fixes a bug, that committed offset is always less than log-end-offset in Kafka by 1 when all messages were processed. This bug causes 2 problems:

  • after the worker restart, the last message from topic partition was re-send to the consumer. (Was fixed by this check in the _drain_messages function: if committed is None or offset > committed. )
  • if there was an offset reset in the partition to the latest, consumer lag is equal to 0. The next message that appears in this partition will be ignored by the consumer due to the previous check, in the _drain_messages function.

This PR increases any committing offset by 1, so the log-end-offset will be equal to 0 when all messages are read.

Fixes #620

@shamanu4
Copy link
Author

Lint errors seem to be unrelated to this PR.

@rochapaulo
Copy link

I am facing this issue, need this fix!

@CLAassistant
Copy link

CLAassistant commented Sep 23, 2020

CLA assistant check
All committers have signed the CLA.

@sivasai-quartic
Copy link

I'm also facing the same issue, need this fix

@patkivikram
Copy link
Collaborator

Can you create this fix on the faust-streaming project? We can merge this to master and create a new release

@leah4kosh
Copy link

hey, any updates on that? we need it :-)

@shamanu4 shamanu4 changed the title Fix committed offset was always less than log-end-offset by 1 Fix committed offset is always less than log-end-offset by 1 Dec 6, 2020
@patkivikram
Copy link
Collaborator

This is already available in faust-streaming

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

First message is ignored after resetting Kafka log offsets to the latest offset

6 participants