feat(eventing): add typed signal-to-event projection architecture - #363
feat(eventing): add typed signal-to-event projection architecture#363robbiemu wants to merge 28 commits into
Conversation
6fb2377 to
2f5ac1c
Compare
2f5ac1c to
0212b99
Compare
# Conflicts: # apps/api/src/services/alerts/AlertsService.ts # apps/api/src/services/integrations/planetscale/PlanetScaleWebhookQueue.ts # apps/cli/src/server/serve.ts # apps/cli/test/server-network.test.ts
|
I now have a working, mostly tested and verified version of this. Just putting a final review / finishing touches on it |
…rting-core # Conflicts: # apps/api/src/services/alerts/AlertsService.ts # apps/cli/src/server/checkpoints.ts # apps/cli/src/server/serve.ts # apps/cli/test/server-network.test.ts
…rting-core # Conflicts: # apps/api/src/services/alerts/AlertsService.ts
|
This is pretty cool and I think this moves alerts into a cool direction and agree with most patterns you build here already tbh! This is quite a massive PR and from looking through it there are quite a lot of small nitpicks/patterns etc. Anyways this looks great and want to get this in asap |
Yes some architectural changes can grow into a bit of a problem, we had to move where and how signals are made into events. One thing we could do if you are not wanting to go through so much manually is to pick archetypical examples to call out, then I can take that input to guide my agent to find and repair similar issues throughout. Not to say I object to you just doing it; it might be the most direct way to get what you want. I'm just happy i don't need to maintain a parallel fork! Happy to get this in. |
Summary
This PR introduces a host-neutral typed signal-to-event projection architecture for Maple’s hosted and Local runtimes, and adds a durable named-consumer boundary to Maple Local.
It separates three concerns:
It also extracts the scheduled-alert decision and delivery policy into a reusable host-neutral package while preserving existing alert behavior.
Related to #222.
Event paths
Immediate per-occurrence path
authenticated input → source adapter → typed normalized signal → bounded selector → pure projector → durable outbox
The original telemetry continues through the existing warehouse encoder. A matched event is staged before the warehouse write and marked ready only after that write succeeds. Retrying the same source occurrence recomputes the same event identity.
Scheduled aggregate path
warehouse query → observation → alert lifecycle evaluation → factual alert event → existing delivery outbox
Rates, thresholds, percentiles, absence, recovery, flap suppression, and renotification remain scheduled conclusions over a window. They are not modeled as individual ingest-time facts.
Core architecture
A source definition publishes a typed field catalog, including allowed operators, sensitivity, and replay capability. Projection configuration stores a bounded typed predicate AST.
Projection revisions compile into immutable registry snapshots only after source fields, operators, activation time, and closed projector configuration are validated. Evaluation runs every matching projection from one snapshot and isolates failures so one malformed projector does not suppress successful siblings.
Projectors are pure, versioned functions. They declare an ID/version, accepted source kinds, output type/schema, and closed configuration decoder. They perform no I/O or external side effects.
Canonical CloudEvents and identity
Projected events use a common versioned CloudEvents envelope.
Event IDs are SHA-256 hashes over a length-delimited tuple of tenant, source kind, source, source occurrence ID, projection ID, and projection revision. Two optional backward-compatible extensions expose source occurrence identity and its quality. Historical envelopes without those extensions remain valid.
This lets downstream consumers correlate source occurrence → immutable Maple event → deterministic transport transaction without parsing event data.
Durable Local outbox and consumers
Maple Local stores projection revisions, active pointers, bounded failures, staged/ready events, and consumer state in a private SQLite control database.
Named consumers support:
Staged events are never pruned. Ready ordering is stable across restart and schema migration. Checkpoint manifests bind the control snapshot alongside the existing data backup.
Alert-core extraction
The new alerting-core package owns host-neutral observation evaluation, trigger/resolve/renotify planning, flap suppression, no-data recovery safety, scheduling helpers, delivery idempotency, and bounded retry policy.
Existing alert queries, persistence, queue behavior, and delivery payloads remain compatible. The factual event envelope is additive.
Existing producer convergence
The existing verified provider-webhook path now creates its factual event through the common projection seam while retaining queue compatibility, including jobs queued before deployment.
This demonstrates the architecture without making any provider-specific vocabulary part of the projection core.
Safety and boundedness
The implementation enforces:
Deliberate boundaries
This PR does not:
Provider adapters, deployment policy, transport delivery, and live credentials remain separate integrations built on the generic contracts introduced here.
Review guide
Primary surfaces:
packages/eventing-core: typed model, predicates, source/projector registries, deterministic identity, schemas, and fixtures;packages/alerting-core: alert evaluation, lifecycle planning, idempotency, scheduling, and retry policy;apps/cli/src/server/eventing: source-neutral normalization, telemetry, runtime, SQLite state, outbox, and consumer protocol;apps/cli/src/server/serve.ts: decode-once integration and authenticated control/consumer endpoints;apps/cli/src/server/checkpoints.ts: eventing-control checkpoint participation;docs/signal-to-event-projection.mdanddocs/local-event-consumers.md; anddocs/eventing-extension-guide.md: a complete compile-time source adapter and projector walkthrough with host wiring, versioning, testing, and review checklists.Review status
Ready for review. Current upstream
mainis merged into the branch, and GitHub reports it mergeable.Validation
Against the clean provider-neutral tree:
main, 146 host-neutral core/Local tests and 103 hosted API tests were rerun successfully;main; andgit diff --checkpasses.