DB error that crashes my node every few minutes

[Ryushka]

What happened?

My node crashed 116 times in less than 24 hours with :

Feb 18 15:32:40 replaced yarn[419870]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const leveldb::Slice&, const leveldb::Slice&): Assertion `r->options.comparator->Compare(key, Slice(r->last_key)) > 0' failed.
Feb 18 15:32:40 replaced yarn[419851]: error Command failed with exit code 1.
Feb 18 15:32:40 replaced yarn[419851]: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Feb 18 15:32:40 replaced yarn[419820]: error Command failed with exit code 1.
Feb 18 15:32:40 replaced yarn[419820]: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Feb 18 15:32:40 replaced systemd[1]: ironfishd.service: Main process exited, code=exited, status=1/FAILURE
Feb 18 15:32:40 replaced systemd[1]: ironfishd.service: Failed with result 'exit-code'.
Feb 18 15:32:50 replaced systemd[1]: ironfishd.service: Scheduled restart job, restart counter is at 140.

Search Logs:
journalctl -u ironfishd -g /deps/leveldb/leveldb-1.20

Feb 17 22:09:00 replaced yarn[292829]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:13:35 replaced yarn[300523]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:17:18 replaced yarn[301087]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:21:56 replaced yarn[301675]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:25:20 replaced yarn[302288]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:28:55 replaced yarn[302777]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:34:46 replaced yarn[303249]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:41:11 replaced yarn[303964]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:49:59 replaced yarn[305047]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 22:59:44 replaced yarn[306779]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:04:47 replaced yarn[308158]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:09:19 replaced yarn[308828]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:16:37 replaced yarn[309594]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:23:05 replaced yarn[310749]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:30:54 replaced yarn[315433]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:37:20 replaced yarn[317685]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:45:39 replaced yarn[319034]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:53:30 replaced yarn[320481]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 17 23:58:30 replaced yarn[321377]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:07:04 replaced yarn[322112]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:14:44 replaced yarn[322953]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:22:33 replaced yarn[325368]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:28:45 replaced yarn[327759]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:35:42 replaced yarn[328994]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:42:25 replaced yarn[331077]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 00:51:02 replaced yarn[331929]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:06:15 replaced yarn[333086]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:12:53 replaced yarn[334040]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:17:43 replaced yarn[335111]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:25:10 replaced yarn[335892]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:33:09 replaced yarn[337085]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:41:42 replaced yarn[338082]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 01:55:08 replaced yarn[339870]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 02:01:53 replaced yarn[342820]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 02:09:48 replaced yarn[343826]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 02:15:02 replaced yarn[344974]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 02:19:46 replaced yarn[345688]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:29:54 replaced yarn[345928]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:37:24 replaced yarn[347534]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:42:05 replaced yarn[348454]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:45:37 replaced yarn[349070]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:49:33 replaced yarn[349605]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:53:17 replaced yarn[350159]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 03:58:11 replaced yarn[350664]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:07:23 replaced yarn[351334]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:12:44 replaced yarn[351966]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:20:03 replaced yarn[352768]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:26:14 replaced yarn[353240]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:38:42 replaced yarn[354954]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:46:38 replaced yarn[355906]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 04:54:35 replaced yarn[356894]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 05:05:46 replaced yarn[357889]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 05:12:17 replaced yarn[359337]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 05:17:43 replaced yarn[360396]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 05:22:25 replaced yarn[361377]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 06:01:28 replaced yarn[361742]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 06:07:25 replaced yarn[363493]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 06:14:59 replaced yarn[364378]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:10:25 replaced yarn[364741]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:18:29 replaced yarn[366894]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:24:34 replaced yarn[367597]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:29:10 replaced yarn[367955]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:35:19 replaced yarn[368686]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:40:42 replaced yarn[369330]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 07:50:23 replaced yarn[370227]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 08:02:04 replaced yarn[371290]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 08:09:28 replaced yarn[372259]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 09:03:16 replaced yarn[373107]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 09:12:07 replaced yarn[373631]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 09:19:57 replaced yarn[374860]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 09:23:32 replaced yarn[375819]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 09:27:57 replaced yarn[376315]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 09:36:36 replaced yarn[377012]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:03:36 replaced yarn[378331]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:07:55 replaced yarn[378695]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:13:37 replaced yarn[379421]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:19:35 replaced yarn[379840]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:26:00 replaced yarn[380503]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:30:52 replaced yarn[381234]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:35:00 replaced yarn[381946]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:39:38 replaced yarn[382651]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:43:25 replaced yarn[383252]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:48:30 replaced yarn[383764]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 10:57:46 replaced yarn[384607]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:03:26 replaced yarn[385979]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:07:11 replaced yarn[386673]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:13:35 replaced yarn[387570]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:18:52 replaced yarn[388534]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:27:43 replaced yarn[389237]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:34:28 replaced yarn[390334]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:38:43 replaced yarn[391191]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:44:22 replaced yarn[391790]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:49:54 replaced yarn[392592]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 11:54:38 replaced yarn[393501]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:09:50 replaced yarn[393819]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:22:19 replaced yarn[394885]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:28:22 replaced yarn[395993]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:36:03 replaced yarn[396823]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:42:06 replaced yarn[397929]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:47:36 replaced yarn[398647]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 12:57:47 replaced yarn[399401]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:03:30 replaced yarn[400313]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:09:17 replaced yarn[401068]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:14:27 replaced yarn[402046]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:19:26 replaced yarn[402796]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:25:01 replaced yarn[403449]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:33:18 replaced yarn[404251]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:41:20 replaced yarn[405482]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:46:49 replaced yarn[406725]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 13:54:03 replaced yarn[407504]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 14:04:14 replaced yarn[408706]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 14:31:37 replaced yarn[409461]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 15:16:23 replaced yarn[410314]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 15:21:56 replaced yarn[417992]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 15:28:41 replaced yarn[418689]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
Feb 18 15:32:40 replaced yarn[419870]: node: ../deps/leveldb/leveldb-1.20/table/table_builder.cc:97: void leveldb::TableBuilder::Add(const >
lines 124-157/157 (END)

Version

0.1.67

Debug output

No response

Relevant log output

Iron Fish version               0.1.67 @ src
Iron Fish library               0.0.44 @ src
Operating system                Linux x64
CPU model(s)                    Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
CPU threads                     16
RAM total                       15.52 GiB
Heap total                      3.99 GiB
Node version                    v18.14.0
ironfish in PATH                true
Garbage Collector Exposed       true
Telemetry enabled               true
Node name                       PaRUSaI
Block graffiti                  PaRUSaI
Done in 2.22s.

Graffiti

PaRUSaI