OutboundSender stops sending events to a webhook

[kristinapathak]

Occasionally, an outbound sender cuts off an endpoint and somehow manages to get into a state where it no longer sends any events to that endpoint ever again for the duration of the outbound sender's existence. The outbound sender receives events and queues them, but no events are ever taken off the queue and attempted to be sent. This leads to the queue slowly filling up and then being cleared when the endpoint is cut off.

The graph of the outgoing_queue_depths for the specific url will look like this:

and the delivery_count metric for that url is always 0.

I believe the root cause of this issue lies somewhere either with how we are emptying the queue:

caduceus/outboundSender.go

Line 440 in 55bce1c

func (obs *CaduceusOutboundSender) Empty(droppedCounter metrics.Counter) {

Or with the dispatcher:

caduceus/outboundSender.go

Line 447 in 55bce1c

func (obs *CaduceusOutboundSender) dispatcher() {