Unhandled

Unhandled messages

Text compliant with Leucine version 0.5.x

Introduction

When you first define your system, it is along the "happy path". Every message that is send will be accepted by the actor. Every accepted message will be processed without error etc. Well, life is not always like that. The moments the message can leave the happy path are discussed here. And there are several. We will assume though that the problem does not occur at message construction. Since that is, after all, fully in your own hands.

Kinds of unhandled messages.

After the message has been constructed it is send to an actor with send(..) or with the tell operator: !. In both cases the receiving actor will try to put it into its mailbox. This can fail for two reasons:

• The actor has already been stopped by Stop.Direct. Or it may still be processing the current mailbox, but does not accept any new messages because of a stop call Stop.Finish. Either way, the message is refused.
• The mailbox is full. Note, "full" means (per default) that there are Int.MaxValue messages in there. Most likely your system already experienced some other issues. But this limit may also be lower, when you make use of your own system parameters. (Note that use of the ProtectAid with a protectLevel does not give rise to refused messages. This will only issue a warning for you to act upon.)

In both cases the send method, with has the (actor dependent) signature:

/**
 * Send a letter, with the option to say who is sending it. Defaults to anonymous outside the context
 * of an actor and to self inside an actor. Returns if the letter was accepted for delivery. Note, this
 * does not mean it also processed. In the mean time the actor may stop. */
def send(letter: Letter, sender: Sender): Boolean

will return false. So a proper designed actor system will have to check that return value, and act accordingly. In every day life, this is a bit of a burden (and it is not possible with the tell operator anyway). So there is an other solution, see below.

Assuming the message was accepted and arrives in the mailbox for later processing, the next thing that can go wrong is that the actor is stopped, before the letter is processed, for example, because the actor is stopped from the outside with Stop.Direct. In that case the stopped(..) callback, which has the signature:

  /**
   * Called before actor deactivation and guaranteed after the last message is processed. If there were
   * any unprocessed messages in this actor at tear down, complete is false. These could be in the normal
   * mailbox or on the stash, if present. Cause returns the last stop mode, so the cause of stopping
   * this actor is known. In case of a actorContext shutdown this is NOT called, for this disruptively
   * terminates all processing loops. It is however called when stop(...) is used, or when the actor
   * is shutdown by a parent. The actor may still be around after this method is called, but will never
   * accept new messages. The parent is still defined, when stopped() is executed (but may already
   * stopped processing messages) but all the children will already be removed from the list, and their
   * stopped() methods have already been called. Apart from the situation described above, you can rely
   * on started() and stopped() to always come in pairs, even when no messages are processed at all. */
  protected def stopped(cause: Actor.Stop, complete: Boolean): Unit = ()

will be called with complete=false. You will know that there were unprocessed messages, but which, that cannot be discerned from this limited information. This is because then callback stopped(..) should not be used to secretly finish the unprocessed letters. The parameter complete is meant for you as indication there may be actions hanging, and to act accordingly. Again, in every day life, this is a bit of a burden, so there is an other solution, see below.

Now let us assume further the actor is left alone and the message finally arrives at the receive(..) method with the (actor dependent) signature:

/**
 * Implement this method in your actor to process the letters send to you. There sender contains a reference
 * to the actor that send the message. To be able to return an answer, you must know the original actor type.
 * This can be obtained by a runtime type match. Use the send method on the senders matched type.  */
protected def receive(letter: Letter, sender: Sender): Receive

and is split into the letter and sender for matching. It may happen that the matching is incomplete. Just letting this 'be' is of course a fatal error for which Leucine cannot protect you. If the compiler is unable to verify that all cases have been handled (this may sometimes happen, even if you use sealed trait, you should use the catch all with unmatched(..) like this

def receive(letter: Letter, sender: Sender): Receive = (letter,sender) match
  ... /* All your cases */
  case _  => unmatched(letter,sender)

This way you do not get a runtime error, and if everything is well, that code will never be executed. But in the unlike case it will, the message is caught for you, and saved for later inspection, see below.

If you make use of stateful actor, you must end the handler with the new state you want to be used. Usually that will be just the old state, thus:

def receive(letter: Letter, sender: Sender): Receive = (state: State) => (letter,sender) match
  ... /* All your cases */
  case _  => unmatched(letter,sender); state

But of course this does not happen ;) and the letter gets processed by one of your match cases. This still can go haywire, and cause an exception or error. First of all, Leucine is not able to catch a Java Error, even if not fatal. If such behavior is expected, please try .. catch them using custom code. If not, this causes the execution to halt. But it a Java Exception occurs, it is caught by the processLoop executor and that calls the except callback method with the (actor dependent) signature:

/**
 * Override this in your actor to process exceptions that occur while processing the letters. The default implementation
 * is to ignore the exception and pass on to the next letter. The letter itself will be filed as Unreadable at the ActorGuard.
 * The size is the total number of exceptions this actor experienced. You may decide to:
 * (1) Stop the actor, by calling stop(Actor.Stop.Direct) inside the handler.
 * (2) Continue for all or certain types of exceptions.
 * (3) Inform the parent if part of a family...
 * This can all be defined in this handler, so there is no need to configure some general actor behavior. If actors
 * can be grouped with respect to the way exceptions are handled, you may define this in your CustomAid mixin, for
 * example, just log the exception. Runtime errors cannot be caught and bubble up. */
protected def except(letter: Letter, sender: Sender, cause: Exception, size: Int): Unit

If you do not implement this method, the message exception is filed as unhandled, and saved for later inspection, see below. If you do, the processing of this letter is completed as far as Leucine is concerned.

To conclude, we do expect the letter to be processing without exceptions of course. This also completes the chain of events for Leucine. There are no more cases in which a problem can arise, a complete system failure left aside.

The stages in which the message can be are captured in the enumerated Actor.Mail:

/**
 * These are the stages a mail can be in. There can be a few reasons why mail is not handled by
 * the actor. Those will be collected in a separate list for manual inspection. The reason is
 * specified by one of these causes. Otherwise we have the regular stages: Empty, Received and
 * Processed. */
enum Mail extends EnumOrder[Mail] :
  /** The mail is not yet defined. */
  case Empty
  /** The mail was not delivered to the actor because it has stopped. */
  case Undelivered
  /** The mail was not accepted by the actor because the mailbox is full. */
  case Unaccepted
  /** The mail was received by the actor for processing. */
  case Received
  /** The mail was not matched by the actor because the type is unknown. */
  case Unmatched
  /** The mail was not read completely by the actor because of an unhandled exception. */
  case Unreadable
  /** The mail was not processed by the actor because it stopped prematurely. */
  case Unprocessed
  /** The mail was processed by the actor, this completes the handling. */
  case Processed

which is part of the returned Actor.Post type that is send to you for each unhandled message.

Catching the unhandled message.

Any unhandled message, whatever the reason is counted as failed which is a metric that is collected by the MonitorAid. In ideal circumstances, this number should be zero over the entire lifetime of the actor.

There is however an other way in which you can see which letters are unhandled. You can register a Actor.Post handling function on the ActorGuard. This function will be called on every occasion a letter could not be handled by the actor system. In ideal circumstances, this should be very seldom of never at all. The idea is that you couple this to you logging system with a critical level of severity. It is also possible to designate a special actor that handles this, for example, by sending the system admin a text message.

The implementation of Actor.Post is:

/** Class to capture the letter, sender en receiver in string form for manual analysis */
case class Post(val mail: Mail, val receiver: String, val letter: String, val sender: String) extends Ordered[Post] :
  def compare(that: Post): Int = ...
  def short: String = s"from=$sender, to=$receiver, letter=$letter"
  def full: String  = s"from=$sender, to=$receiver, mail=$mail, letter=$letter"

so it has some basis display methods, but you can of course also define others. The signature of ActorGuard.failed(..) is:

/**
 * Here you register a handler for failed messages. This includes all messages
 * that are fused at sending, cause an exception that is not catched by the user,
 * is unmatched (manually added) or is left over as unprocessed when the actor
 * stops prematurely. The post handler can be called in any context, so the
 * execution should be very brief. It is for logging, or post processing by
 * sending it to an other actor. Nothing more. Define this handler at the
 * beginning of your application and make sure it is only defined once.
 **/
def failed(post: Actor.Post => Unit): Unit

If you send this information to your logger with WARN level this resembles the Akka system for reporting dead letters. For example:

ActorGuard.failed(post => logger.warn(post.full))