Hi! The roadmap under Call for Contributors lists "data storage &
processing", with a sqlite schema + tools as a possible starting point,
and #566 has discussion along those lines (thebentern: dirt-simple sqlite
makes sense because it's portable and queryable). I'd like to propose a
concrete minimal shape and hear whether this is wanted before I open a PR.
The gap I kept hitting: --listen shows packets once and forgets them.
Third-party tools keep their own history instead (MeshMonitor has a full db,
pdxlocations/contact stores messages in its own sqlite), so every tool
reinvents storage differently. On top of that the device-side queue only
holds ~30 packets, so whatever a client misses while disconnected is gone
unless an S&F server happens to be running.
What I've been running locally while testing (works well):
- meshtastic/recorder.py: a small Recorder class subscribed to
"meshtastic.receive"
- one new flag: --record [FILE] (defaults to meshtastic.db), stays open
like --listen, clean close on Ctrl-C
- small versioned SQLite schema (PRAGMA user_version), stdlib only, no
new deps; stores a numeric sender id (from_num) so identities learned
later still resolve onto older rows:
packets(id PK, rx_time, from_id, from_num, to_id, channel, portnum,
text, snr, rssi, hop_limit, packet_id UNIQUE)
nodes(node_num PK, user_id, long_name, short_name, hw_model,
last_heard) <- upserted from NODEINFO packets, so stored history reads
like a chat instead of hex IDs
- WAL + busy_timeout, INSERT OR IGNORE dedup by packet_id, per-packet
commit (crash safe)
- 17 offline pytest cases: storage, dedup by packet id, NODEINFO name
resolution, a cross-thread write regression, plus coverage for the
history reader (ordering, limits, missing-db errors)
Live tested against hardware: 48 packets captured in a 3 minute window on
a busy EU_868 net with zero write failures, node names auto-learned, and
device-queue replays after reconnect land in the db automatically (they
arrive through the same pubsub path).
Questions before I open a PR:
- does a minimal packets/nodes schema like this match what you'd want
standardized, or should it record raw protobuf blobs too?
- is --record the flag name you'd prefer, placed next to --listen?
- want a read side later (--history last N / filters) as a separate
small PR?
Happy to open the PR against master with tests included if this sounds
welcome.
Hi! The roadmap under Call for Contributors lists "data storage &
processing", with a sqlite schema + tools as a possible starting point,
and #566 has discussion along those lines (thebentern: dirt-simple sqlite
makes sense because it's portable and queryable). I'd like to propose a
concrete minimal shape and hear whether this is wanted before I open a PR.
The gap I kept hitting: --listen shows packets once and forgets them.
Third-party tools keep their own history instead (MeshMonitor has a full db,
pdxlocations/contact stores messages in its own sqlite), so every tool
reinvents storage differently. On top of that the device-side queue only
holds ~30 packets, so whatever a client misses while disconnected is gone
unless an S&F server happens to be running.
What I've been running locally while testing (works well):
"meshtastic.receive"
like --listen, clean close on Ctrl-C
new deps; stores a numeric sender id (from_num) so identities learned
later still resolve onto older rows:
packets(id PK, rx_time, from_id, from_num, to_id, channel, portnum,
text, snr, rssi, hop_limit, packet_id UNIQUE)
nodes(node_num PK, user_id, long_name, short_name, hw_model,
last_heard) <- upserted from NODEINFO packets, so stored history reads
like a chat instead of hex IDs
commit (crash safe)
resolution, a cross-thread write regression, plus coverage for the
history reader (ordering, limits, missing-db errors)
Live tested against hardware: 48 packets captured in a 3 minute window on
a busy EU_868 net with zero write failures, node names auto-learned, and
device-queue replays after reconnect land in the db automatically (they
arrive through the same pubsub path).
Questions before I open a PR:
standardized, or should it record raw protobuf blobs too?
small PR?
Happy to open the PR against master with tests included if this sounds
welcome.