fix(queues): continue processing after hard error in handler

[sabertobihwy]

Summary

There was an issue with the queue implementation where processing would stop when encountering a hard error in the handler function. The changes in the code fix this problem.

Now, the handler function is called using pcall to catch any errors that occur. If a hard error occurs, the error is logged but the processing of the remaining entries in the queue continues.

Checklist

• The Pull Request has tests
• There's an entry in the CHANGELOG
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

• [Implement ...]

Issue reference

Fix KAG-2361

[hanshuebner]

Instead of simply continuing as if the call succeeded, we should treat the error like one that was returned by the handler. This will cause the call to be retried according to the retry policy, which is preferable in my opinion.

[sabertobihwy]

updates

[hanshuebner]

There is no need for a nested conditional. If the ok is true, then log the debug message and break. Otherwise, log a message that indicates whether the handler returned a failure or it failed executing altogether (with the respective error message). Also remove the comment "We just log the error and continue, since we want to continue executing the remaining entries in the queue." because it no longer reflects what the code does.

[sabertobihwy]

update

[hanshuebner]

There is no need for a nested conditional. If the ok is true, then log the debug message and break. Otherwise, log a message that indicates whether the handler returned a failure or it failed executing altogether (with the respective error message). Also remove the comment "We just log the error and continue, since we want to continue executing the remaining entries in the queue." because it no longer reflects what the code does.

[hanshuebner]

I would have committed the changes myself, but as you're not working off the Kong repository but instead use your own fork, I can't easily do that. Replace everything between if status and ok == true then and self:log_warn("handler could not... with:

    if status and ok then
      self:log_debug("handler processed %d entries successfully", entry_count)
      break
    end
    if not status then
      -- protected call failed, ok is the error message
      err = ok
    end

    self:log_warn("handler could not process entries: %s", tostring(err or "no error details returned by handler"))

    if (now() - start_time) > self.max_retry_time then
      self:log_err(
        "could not send entries, giving up after %d retries.  %d queue entries were lost",
        retry_count, entry_count)
      break
    end

[outsinre]

Is this in EE?