Aug26 more improvements

[skunkworker]

Correctness fixes for concurrency bugs found while reviewing the 0.13.1 and 0.13.2 changes. No API or configuration changes; version bumped to 0.13.3.pre1.

Every fix ships with a spec that was verified to fail against the pre-fix code — each library file was temporarily reverted to confirm the spec caught the bug, not just that it passed afterward. Suite goes from 287 to 297 examples, 0 failures, and the two pre-existing RSpec false-positive warnings are gone.

The one that would have caused an outage

ByteBoundedQueue byte counter drifted permanently upward (341864a)

push counted an item's bytes after handing it to SizedQueue#push, so a consumer could pop the item and subtract its bytes before the producer had added them. pop clamps at zero, so that subtraction was swallowed, and the producer's increment then landed on an item that was already gone.

The counter therefore only ever ratcheted up. The server funnels every subscription through one shared queue and pops it from processor_count handler threads on JRuby, so the race was live on every message. Once the drift reached the 128 MiB ceiling, push would drop every subsequent request — the server goes dark while still looking healthy. This is the same failure class as the nats-pure pending_size drift fixed in 0.13.1.

Bytes are now counted before the enqueue and rolled back in an ensure if the enqueue doesn't happen. The regression spec fails on the old code with 100 phantom bytes in an empty queue.

Shutdown

Work accepted as shutdown begins was stranded (fbce03b, b912e5c)

ThreadPool#push checks the shutdown flag and then enqueues, so shutdown can slip its poison pills between those two steps. Workers took a pill and exited, leaving the work behind them unrun — and the server has already published an ACK for that request, so its client blocked until response_timeout (60s default). The abandoned @active_work increment leaked too.

Re-checking the flag after the increment was the obvious fix, but it doesn't close the window and introduces a false negative: work enqueued before the pills runs regardless, so reporting it as rejected would make the caller NACK work that executes anyway — duplicating effects for non-idempotent RPCs. Instead, workers drain any work queued behind their pill (putting a sibling's pill back so every worker still exits).

That alone turned out to be insufficient, which surfaced as a ~8% flake in the new specs — a real hole in the fix, not test timing. If a worker takes its pill and drains before the racing push lands, it finds an empty queue and exits, stranding the work exactly as before. wait_for_termination now drains once more after the last worker is gone, on the caller's thread where nothing can race it. Admission and shutdown still aren't atomic (#push is lock-free by design), but nothing enqueued before the pool reports termination is dropped.

Timeout.timeout(10) reintroduced the JRuby mutex hazard (bc504e4)

0.13.1 removed Timeout from SuperSubscriptionManager because its async Thread#raise, fired while a thread holds the SizedQueue mutex, makes JRuby unwind through the held mutex and raise "Attempt to unlock a mutex which is locked by another thread/fiber". Server#run still wrapped the whole shutdown call in Timeout.timeout(10), reintroducing the hazard one frame up.

It could genuinely fire: #shutdown is not bounded by 10s. Worst case is one 1s push deadline per handler, then a 5s join, then 1s kill-joins — past 10s once there are more than a few handlers, and the JRuby default is processor_count. The resulting ThreadError is caught by the existing rescue, but the corrupted queue mutex can then hang the trailing @pending_queue.clear.

#shutdown already self-bounds with a monotonic deadline and non-blocking pushes, so the wrapper is gone, along with the now-dead require "timeout" in both files. A guard spec fails if it comes back.

Self-healing

A late muxer dispatcher tore down a subscription a sibling had rebuilt (9b8eb2e)

A dispatcher that crashes fatally sleeps a backoff, then unconditionally called drop_subscription_locked. When several crash together (JRuby runs processor_count of them) they wake on different backoffs — 1s, then 4s — so the late one destroyed the subscription the earlier one had just rebuilt, and fail_inflight_requests cancelled every request that had already arrived on it. The staggered backoff makes this more likely, not less.

Each dispatcher now tears down only the subscription it actually died on. Worth noting for review: capturing that subscription when the thread starts does not work — it races the sibling's swap and reads whichever subscription is current after the backoff, which is the very value the check needs to compare against. The dispatch loop records what it's draining in a thread-local instead.

Observability

Non-UUID tokens reported ~56-year message ages (f2799c8)

String#to_i(16) stops at the first non-hex character and returns 0 rather than raising, so extract_timestamp's length-only check let a foreign reply token parse as epoch 0. age_in_seconds then reported ~1.79e9 seconds into the client.unexpected_message gauge, skewing dashboards.

extract_timestamp now validates the whole token. Impact is metrics quality, not correctness — in practice these tokens are this gem's own UUIDv7s, so a garbage age needs a foreign publisher on the inbox subject. Validation accepts both the dashed and compact forms, since the dashed regex alone would have failed the existing "handles UUIDs without dashes" spec.

Also switches two not_to raise_error(NoMethodError) matchers to the bare form; RSpec warns those pass on any error, including one raised before the code under test is reached.

Not addressed here

• TLS hostname (SAN/CN) verification — still off, tracked separately. Chain verification landed in 0.13.1.
• Per-message sub.synchronize in the muxer dispatch loop — a documented throughput trade for the byte cap, not a defect. Worth measuring before changing.
• Dead SystemExit/Interrupt guard inside a StandardError rescue in super_subscription_manager.rb:268 — cosmetic.

Verification

• JRuby 9.4.14.0 (3.1.7), 297 examples, 0 failures, warning-free, stable across five consecutive full runs.
• The previously-flaky drain spec passes 25/25 after the follow-up fix.
• Gem builds cleanly as 0.13.3.pre1.