Fix ForeignKeyViolation crash when rescheduling sensors during failure - #71966
Open
saitejabandaru-in wants to merge 1 commit into
Open
Fix ForeignKeyViolation crash when rescheduling sensors during failure#71966saitejabandaru-in wants to merge 1 commit into
saitejabandaru-in wants to merge 1 commit into
Conversation
This moves the `prepare_db_for_next_try` call in `fetch_handle_failure_context` to execute after the listener hook (`on_task_instance_failed`). Previously, a slow listener could extend the window between deleting old `task_reschedule` rows and assigning a new `ti.id`, allowing a concurrent reschedule insert (using the old `ti.id`) to arrive and block the subsequent `ti.id` update flush with a ForeignKeyViolation. This closes the gap to microseconds and ensures listeners see the correct (old) `ti.id` that actually failed. Fixes apache#71923.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #71923.
Motivation
When a sensor fails, the scheduler calls
TaskInstance.fetch_handle_failure_context. This method clears oldertask_reschedulerecords and mutatesti.idto a new UUID by callingti.prepare_db_for_next_try, and then it callson_task_instance_failedlistener hooks before flushing the updatedti.idto the database.If a listener hook blocks for seconds (e.g. OpenLineage timing out), the transaction window is extended. Concurrently, an API server might process a reschedule request from the supervisor and insert a new
task_reschedulerow using the oldti.id. When the listener finishes, the scheduler flushesUPDATE task_instance SET id = <new_id> WHERE id = <old_id>. But since the newly insertedtask_reschedulepoints toold_id, PostgreSQL rejects the update with aForeignKeyViolation, causing the scheduler to crash.Changes
ti.prepare_db_for_next_try(session)infetch_handle_failure_contextto run after the listener hooks.task_rescheduledelete and theti.idupdate flush from seconds down to microseconds.TaskInstancewith the old (actual) UUID that experienced the failure, rather than the prematurely rolled new UUID that hasn't run yet.