[v3-3-test] Reuse the ambient session when skipping pending TIs in dag-run terminal state (#71488) - #71955
Draft
github-actions[bot] wants to merge 1 commit into
Draft
[v3-3-test] Reuse the ambient session when skipping pending TIs in dag-run terminal state (#71488)#71955github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
…g-run terminal state (#71488) TaskInstance.set_state is @provide_session: called without a session, every pending task instance gets its own session — a fresh DB connection (with the documented sql_alchemy_pool_enabled=False + pgbouncer setup), a refresh SELECT, a merge SELECT, an UPDATE and a COMMIT. Marking a dag run failed on a run with 3,917 mapped task instances took 280 s (~19,500 statements, ~3,900 connections) on an otherwise idle PostgreSQL 17 with 2.5 ms RTT. Passing the ambient session through drops the per-TI cost from 79 ms to 7.5 ms measured on the same run (x10.5); the dag-run state and the running-TI path already use this session, so transactional semantics are unchanged. (cherry picked from commit 0807f2b) Co-authored-by: Mat <matthieuauger@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TaskInstance.set_state is @provide_session: called without a session,
every pending task instance gets its own session — a fresh DB
connection (with the documented sql_alchemy_pool_enabled=False +
pgbouncer setup), a refresh SELECT, a merge SELECT, an UPDATE and a
COMMIT. Marking a dag run failed on a run with 3,917 mapped task
instances took 280 s (~19,500 statements, ~3,900 connections) on an
otherwise idle PostgreSQL 17 with 2.5 ms RTT.
Passing the ambient session through drops the per-TI cost from 79 ms
to 7.5 ms measured on the same run (x10.5); the dag-run state and the
running-TI path already use this session, so transactional semantics
are unchanged.
(cherry picked from commit 0807f2b)
Co-authored-by: Mat matthieuauger@users.noreply.github.com
Co-authored-by: Claude Fable 5 noreply@anthropic.com