Skip to content

[AMORO-4348] Fix infinite retry loop for failed table processes - #4349

Open
j1wonpark wants to merge 1 commit into
apache:masterfrom
j1wonpark:fix/process-infinite-retry-loop
Open

[AMORO-4348] Fix infinite retry loop for failed table processes#4349
j1wonpark wants to merge 1 commit into
apache:masterfrom
j1wonpark:fix/process-infinite-retry-loop

Conversation

@j1wonpark

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Fix #4348.

Since #4116, ProcessService constructs DefaultTableProcessStore passing the current retry count into the constructor's maxRetryTime parameter. A new process therefore gets maxRetryTime = 0 and becomes terminal on its first failure, every subsequent transition is rejected, and retryNumber never increases — while the retry branch in executeOrTraceProcess keeps evaluating retryNumber < PROCESS_MAX_RETRY_NUMBER as true and resubmits the process forever, with no backoff.

Brief change log

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

TestDefaultProcessService#testFailedProcessRetryIsBounded reproduces the infinite loop before the fix (the process is never dropped and the wait times out) and passes after: the failing process is submitted 1 + PROCESS_MAX_RETRY_NUMBER times in total and then untracked. Existing process service tests all pass.

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)

Signed-off-by: Jiwon Park <jiwonpark@apache.org>
@github-actions github-actions Bot added the module:ams-server Ams server module label Aug 31, 2026
@xxubai xxubai added this to the Release 0.9.0 milestone Sep 1, 2026

@Override
public ProcessStatus getStatus(String processIdentifier) {
if (failedIdentifiers.contains(processIdentifier)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Please make the fake engine fail directly from submitTableProcess().

Returning FAILED immediately takes the fast-terminal path and produces redundant COMPLETE_FAILED transition errors.

Suggested change
if (failedIdentifiers.contains(processIdentifier)) {
throw new IllegalStateException("Submission failure");

@zhoujinsong zhoujinsong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Failed table process is resubmitted in an infinite retry loop

3 participants