UN-3016 [FIX] Record why an execution failed; stop dispatching skipped files - #2256
Open
hari-kuriakose wants to merge 7 commits into
Open
UN-3016 [FIX] Record why an execution failed; stop dispatching skipped files#2256hari-kuriakose wants to merge 7 commits into
hari-kuriakose wants to merge 7 commits into
Conversation
…d files A customer (Moody's) execution ended in ERROR with no explanation. Verified against the prod row for 30b84e4a-8675-4e93-a235-8d5dbac89be8: status=ERROR, error_message='' (blank), total_files=1. Three distinct defects, fixed here: 1. Execution errors were recorded blank. _determine_execution_status_unified() decided ERROR purely from failure counts and returned no reason, so all three call sites passed error_message=None. WorkflowExecution.update_execution() guards on `if error:`, so error_message was never written. It now returns a reason summarised from the per-file errors already present in aggregated_results, capped to the CharField(256) that otherwise truncates silently. 2. A file skipped for an unsupported MIME type was still handed to a worker. The skip deliberately never wrote the file's bytes, but returned it with is_executed=True and a "temp-hash-<uuid>" sentinel. Nothing downstream filters on is_executed, so a worker ran, failed on the missing file, and produced the opaque "Execution: <path>; Destination: <path>" error seen in the incident row (that string is built by interpolating two FileNotFoundError paths, not by mangling a MIME message). Skipped files are now excluded from the staged set; if every file is skipped the request fails with a 400 naming the unsupported types instead of dispatching an empty execution. 3. The workers' AllowedFileTypes lacked XLSM while the backend had it. The two MIME lists are now identical, so a file the API accepts cannot be rejected again inside the worker. Tests: workers/tests/test_un3016_execution_error.py, 9 passing — covers the real incident error shape, the never-blank guarantee, the 256-char column fit, and a regression guard that no caller reintroduces error_message=None. Note: XLSM was absent from the backend enum at the time of the incident and has since been added, so the customer's .xlsm would be accepted today; the defects above remain for any other unsupported type.
for more information, see https://pre-commit.ci
Remediation of review findings on PR #2256. F1 (High): the new UnsupportedMimeTypeError raised by add_input_file_to_api_storage escaped the try/except in WorkflowViewSet.execute that owns delete_api_storage_dir, so a partial stage could leave written files behind. The staging call now has its own handler mirroring the one at the end of the method. The sibling caller (api_v2/deployment_helper.py:279) was already guarded. F2 (High): test_no_caller_passes_a_hardcoded_none_error asserted "error_message=None" not in the whole 1900-line tasks.py. It matched nothing at the sites it meant to guard and would trip on any unrelated keyword default. Now an AST check scoped to _process_batch_callback_core and process_batch_callback_api. Mutation-checked: reintroducing error_message=None at process_batch_callback_api fails the test with that call site's line number. F4 (Medium): _EXECUTION_ERROR_MAX_LENGTH now names its authority, EXECUTION_ERROR_LENGTH in backend workflow_v2/models/execution.py, following the convention in workflow_v2/undispatched_sweep.py. Adversarial verification also found two comments I had written asserting mechanisms the code does not have (the storage dir is a computed path, not a mkdir; the AST check does not catch positional/indirected None). Both false claims deleted rather than reworded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQhqNqCwFcXZZ7cQU6HxUE
…rce shape Behaviour-preserving cleanup, run once after the review verdict settled. - views.py: drop the second cleanup handler added in the previous commit and stage inside the existing try instead. That handler's guard is exactly the staging condition, so one handler now covers both paths rather than two copies of the same contract. - test_un3016_execution_error.py: import callback.tasks directly instead of extracting the helper with ast/exec. The docstring's claim that importing pulls in an unusable celery runtime is false — conftest loads .env.test before collection, and test_pg_callback_duplicate_guard.py already imports the module at module level. Verified by running the import under pytest. test_status_function_returns_a_reason is now behavioural: it calls _determine_execution_status_unified and asserts the reason is non-blank. The old version asserted only that every return was a 4-tuple, which would have passed with an always-None fourth element — i.e. it could not detect the very defect it was named for. Mutation-checked: forcing error_message = None now fails the test. - _summarize_file_errors: errors is keyed by file name, so entries are distinct by construction and the `entry not in seen` dedup could never fire. Removed, along with the duplicate early return it guarded. - source.py: skipped_files was a dict never used as a mapping; now a list of pre-formatted entries. Tests: 1298 passed, 132 skipped. test_pg_reaper.py deselected — it needs a live Postgres on 127.0.0.1:5432 and hangs identically on unmodified HEAD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQhqNqCwFcXZZ7cQU6HxUE
_determine_execution_status_unified marks ERROR from two places, and a blank error_message from either one is the UN-3016 defect. Only the failed_files == total_files branch was covered; the timeout branch (files expected, no batch result came back) had no test at all — mutating its error_message to None left the suite green. Adds test_timeout_failure_also_returns_a_reason, which mocks the api_client so get_workflow_execution reports total_files=3 with an empty file_batch_results, driving has_timeout_failure. Mutation-checked: forcing that branch's error_message to None now fails the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQhqNqCwFcXZZ7cQU6HxUE
Contributor
Author
Note: partial file skips are silent to the API caller — deliberately not fixed hereFlagged during review and waived, not overlooked. Recording the reasoning so it is stated rather than rediscovered later. The behaviour. The guard added in this PR is at if skipped_files and not file_hashes:
raise UnsupportedMimeTypeError(
"No files could be processed. Unsupported file type(s): " + ", ".join(skipped_files)
)The Why it is not fixed in this PR.
The asymmetry worth knowing about. Total failure is now loud and explicit; partial failure is quiet. That is a deliberate trade, not an oversight — but it does mean a user can get a successful run over fewer files than they uploaded without the response saying so. Suggested follow-up: report skipped files to the caller, e.g. as a Reviewed with unstract:lite-remediation; finding F3, waived. |
PR #2256's workers half had 8 tests; its backend half had none. Six tests close that gap, all DB-free (the ORM boundary and file storage are patched, so no Postgres is needed). source.py — add_input_file_to_api_storage: - partial skip returns only the supported files, and does NOT raise. A rejected file alongside a runnable one is simply absent from the mapping; the request proceeds with what can be run (UN-4055 tracks whether that should remain the behaviour, so this pins it rather than asserting a raise). - a total skip raises UnsupportedMimeTypeError naming every skipped file and its MIME type, instead of dispatching an empty execution that would report a vacuous success. - an empty request returns {} without raising: the guard is on something having been skipped, not merely on the mapping being empty. - an accepted file's FileHash carries the sha256 of the staged bytes. This is characterisation only — it held before the fix and no mutation of the fix makes it fail; it documents what a returned entry looks like, which is what makes the rejected file's absence meaningful. views.py — WorkflowViewSet.execute: - a staging failure reaches delete_api_storage_dir, so a partial stage does not leave written files behind. - a request that staged nothing does not attempt that cleanup. Files are real SimpleUploadedFile objects, not mocks: a MagicMock's content_type is never in AllowedFileTypes, so a mocked "supported" file would silently take the skip branch and the test would pass for the wrong reason. Mutation-checked, each mutation reverted and the restore verified on disk: - re-adding the rejected file to file_hashes with a temp hash -> partial-skip and total-skip tests fail - dropping "and not file_hashes" from the raise condition -> partial-skip test fails (it now raises on a request that has runnable files) - dropping "skipped_files and" from the raise condition -> empty-request test fails - deleting the raise block -> total-skip test fails - stripping the filenames from the error message -> total-skip test fails - moving staging back outside the try -> staging-failure test fails - dropping the has_uploads guard on cleanup -> no-cleanup test fails Five of the six tests are pinned by a mutation; the sha256 test is the characterisation case noted above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQhqNqCwFcXZZ7cQU6HxUE
hari-kuriakose
marked this pull request as ready for review
August 31, 2026 19:48
|
Contributor
|
| Filename | Overview |
|---|---|
| backend/workflow_manager/endpoint_v2/source.py | Filters unsupported uploads out of the staged mapping and raises a descriptive error when no processable files remain. |
| backend/workflow_manager/workflow_v2/views.py | Moves upload staging into the existing cleanup boundary so staging failures remove execution-scoped API storage. |
| workers/callback/tasks.py | Adds bounded execution-error summaries and propagates them through both callback finalization and notification paths. |
| workers/shared/enums/file_types.py | Adds the XLSM MIME type to align worker validation with backend acceptance. |
Sequence Diagram
sequenceDiagram
participant C as API client
participant B as Backend staging
participant W as File worker
participant K as Callback
participant E as Execution record
C->>B: Upload files
B->>B: Filter unsupported MIME types
alt No supported files
B-->>C: 400 with unsupported types
else Supported files remain
B->>W: Dispatch only staged files
W-->>K: Per-file results
K->>K: Determine final status and reason
K->>E: Persist counts, status, and error message
K-->>C: Notify with failure reason when applicable
end
Reviews (1): Last reviewed commit: "Merge branch 'main' into worktree-un3016..." | Re-trigger Greptile
Contributor
Unstract test resultsPer-group results
Critical paths
|
hari-kuriakose
requested review from
chandrasekharan-zipstack and
muhammad-ali-e
August 31, 2026 20:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UN-3016 — an execution failed with no explanation
A customer (Moody's) execution ended in ERROR with nothing telling the user why. Root-caused from code and then corroborated against the actual prod row:
ERROR...Application.xlsm(application/vnd.ms-excel.sheet.macroenabled.12)temp-hash-9d260345dd444238a58eb42ab562bcd7The prod logs for this run are long gone (30-day retention vs a 283-day-old incident), so the DB row is the evidence.
What was happening
The
temp-hash-prefix is written in exactly one place — the backend MIME-skip branch — which pins the mechanism:.xlsmand took the skip branch, which deliberately never writes the file's bytes. But it returned the file withis_executed=Trueand a fake hash.is_executed, so the file was counted intotal_filesand dispatched to a worker anyway.FileNotFoundError(path)stringifies to.failed_files == total_files(1 == 1) escalated the execution to ERROR._determine_execution_status_unifiedreturned only a status and counts — no reason string — so every call site passederror_message=None.WorkflowExecution.update_execution()guards onif error:, soerror_messagewas never written.Verified present at
0da9da5a8, the commit that was HEAD the day the ticket was filed — pre-existing, not a regression.Changes
_determine_execution_status_unifiedreturns a reason built from the per-file errors already aggregated inaggregated_results; all call sites pass it through, including notifications. Capped to theCharField(256)that otherwise truncates silently.AllowedFileTypeswas missing XLSM while the backend had it. The two MIME lists are now identical, so a file the API accepts cannot be rejected again inside the worker.PGMQ
All three apply to the PGMQ stack. It is not a separate callback path — it shares
_process_batch_callback_coreandprocess_batch_callback_apiwithis_pgas a kwarg flag, so both patched call sites cover it. Staging is backend-side and transport-agnostic, and the PG stack has no MIME list of its own. Its own error paths ([pg-poison-drop],[pg-barrier-abort],[reaper-recovery]) already wrote descriptive reasons and never had this defect.Tests
workers/tests/test_un3016_execution_error.py— 9 passing: the real incident error shape, the never-blank guarantee, the 256-char column fit, and a regression guard that no caller reintroduceserror_message=None.Reviewer notes
unstract.corepackage, absent from this checkout. The 3→4 tuple change on_determine_execution_status_unifiedis the thing to watch; the two PG tests touching it mock withassert_not_called, so they should be unaffected, but CI is the real check.FileNotFoundErrorpaths. This PR makes the execution record it rather than a blank — an improvement, not a cure._handle_null_execution_resultalso still publishes nothing to the UI; proven not to be this incident's mechanism, so left alone. Both warrant a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_0197gRFp4QFxu2VzxhTEozGn