A/B firmware & config slots have some non-obvious consequences

[jktjkt]

Our system is designed to use completely separate A and B slots. Each of these slots contain both the root filesystem, as well as a configuration DB. We cannot have a single configuration DB storage partition because some updates change the config schemas, and that is a potentially risky operation -- so we want to have a way of performing rollbacks. The "other" config partition (the one which belongs to a slot that's currently not running) is only written to during a FW installation. At that time, it gets provisioned with the current snapshot of the startup DS. During normal use, the "other" config partition is (deliberately) allowed to drift out-of-sync with the current config partition.

One of the devices which were recently installed did not come back after a power outage. We currently do not know what exactly has happened, but given that the inactive slot's config was not provisioned at all, according to @ares1025, chances are that the system failed over to the other FW slot, and the network did not come back again.

I think that we need "something" to sync the current startup DS to the other FW slot, and "something" to query the current configuration snapshot.

Propagation of startup to the other slot

Right now it only happens via a RAUC hook when the FW gets upgraded. We do not sync it automatically because we originally thought that having a fully separate partition increases robustness. It still does, except when it does not, such as when a system becomes unreachable due to missing network config.

Querying the current status of the "other cfg slot"

There's currently no way to query the content (and other metadata, such as "is this in the same schema version as the current config"?) via standard YANG APIs.

Possible implementation

A custom DS

We could add a custom DS to sysrepo (somehow), and maybe only allow the copy-config operation. Except that:

• there's no obvious support for "simply adding a new and limited DS to all installed modules" in sysrepo, AFAICT
• there's no copy-config in RESTCONF, so it likely means listening for a complete replace operation, or, worst case, semi-arbitrary writes
• the config lives on an unmounted FS

Explicit actions via the czechlight-system YANG model

We could add some ad-hoc actions which inspect the content of the unmountd FS (maybe via debugfs), and an action that re-runs the config installation hook or otherwise synces the other config slot with the current startup DS..