Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix committed offset is always less than log-end-offset by 1 #634

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

shamanu4
Copy link

Description

This PR fixes a bug, that committed offset is always less than log-end-offset in Kafka by 1 when all messages were processed. This bug causes 2 problems:

  • after the worker restart, the last message from topic partition was re-send to the consumer. (Was fixed by this check in the _drain_messages function: if committed is None or offset > committed. )
  • if there was an offset reset in the partition to the latest, consumer lag is equal to 0. The next message that appears in this partition will be ignored by the consumer due to the previous check, in the _drain_messages function.

This PR increases any committing offset by 1, so the log-end-offset will be equal to 0 when all messages are read.

Fixes #620

@shamanu4
Copy link
Author

Lint errors seem to be unrelated to this PR.

@rochapaulo
Copy link

I am facing this issue, need this fix!

@CLAassistant
Copy link

CLAassistant commented Sep 23, 2020

CLA assistant check
All committers have signed the CLA.

@sivasai-quartic
Copy link

I'm also facing the same issue, need this fix

@patkivikram
Copy link
Collaborator

Can you create this fix on the faust-streaming project? We can merge this to master and create a new release

@leah4kosh
Copy link

hey, any updates on that? we need it :-)

@shamanu4 shamanu4 changed the title Fix committed offset was always less than log-end-offset by 1 Fix committed offset is always less than log-end-offset by 1 Dec 6, 2020
@patkivikram
Copy link
Collaborator

This is already available in faust-streaming

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

First message is ignored after resetting Kafka log offsets to the latest offset
6 participants