Skip to content

[fix](load) Respect Kafka isolation level when fetching offsets - #67089

Open
sollhui wants to merge 1 commit into
apache:masterfrom
sollhui:agent/routine-load-isolation-aware-offsets
Open

[fix](load) Respect Kafka isolation level when fetching offsets#67089
sollhui wants to merge 1 commit into
apache:masterfrom
sollhui:agent/routine-load-isolation-aware-offsets

Conversation

@sollhui

@sollhui sollhui commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Routine load used query_watermark_offsets() when resolving OFFSET_END and refreshing latest offsets. The high watermark is the log end offset and includes records from open Kafka transactions, so isolation.level=read_committed did not affect routine load progress, lag, or scheduling decisions.

This change uses Kafka’s ListOffsets Admin API and passes the job isolation level explicitly. read_committed now returns the last stable offset, while read_uncommitted continues to return the log end offset. Both periodic latest-offset refresh and OFFSET_END resolution share the same implementation.

A transactional Kafka regression case covers the different initial offsets and lag values for read_committed and read_uncommitted jobs.

Release note

Kafka routine load offset queries now respect property.isolation.level.

Check List (For Author)

  • Test: Not run (skipped at requester direction)
  • Behavior changed: Yes. read_committed offset and lag exclude records in open transactions.
  • Does this need documentation: No

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Routine load queried Kafka watermark offsets for latest offsets and OFFSET_END. Watermark queries always expose the log end offset, so jobs configured with isolation.level=read_committed still counted records from open transactions. Use the Kafka ListOffsets Admin API with the job isolation level so read_committed returns the last stable offset while read_uncommitted keeps using the log end offset. Add a transactional Kafka regression case covering initial OFFSET_END resolution and lag refresh.

### Release note

Kafka routine load offset queries now respect property.isolation.level.

### Check List (For Author)

- Test: Not run (skipped at requester direction)
- Behavior changed: Yes (read_committed offset and lag exclude open transactions)
- Does this need documentation: No
@sollhui
sollhui force-pushed the agent/routine-load-isolation-aware-offsets branch from b8d1b8d to ac58348 Compare August 25, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants