Extension to restore desired state on deviceAnnounce

[sjorge]

/*
 * This extension will restore a predefined state to a device on deviceAnnouncement
 *
 * In `configuration.yaml` under the device add a `restore_state` object, per-device
 * settings will overwrite device_options if both have the same property.
 *
 * device_options:
 *   debounce: 1
 *     restore_state:
 *       state: OFF
 * devices:
 *   '0x000d6ffffe8e8d4f':
 *     friendly_name: masterbedroom/lamp
 *     restore_state:
 *       state: ON
 *       color: "#FF0000"
 *
 */
class Restore {
    constructor(zigbee, mqtt, state, publishEntityState, eventBus, settings, logger) {
        this.zigbee = zigbee;
        this.mqtt = mqtt;
        this.state = state;
        this.publishEntityState = publishEntityState;
        this.eventBus = eventBus;
        this.settings = settings;
        this.logger = logger;
    }

    async start() {
        this.logger.info('Starting device state restore');

        // retrieve shared restore_state
        const device_options = this.settings.get().device_options;
        this.shared_restore_state = device_options.hasOwnProperty("restore_state") ? device_options.restore_state : {};

        // attach to events
        this.eventBus.on('deviceAnnounce', this.onDeviceAnnounce.bind(this), this.constructor.name);
    }

    async stop() {
        this.eventBus.removeListeners(this.constructor.name);
    }

    async onDeviceAnnounce(data) {
        const device = this.zigbee.resolveDevice(data.device.ieeeAddr);

        // abort early if we are interviewing
        if (!device.zh.interviewCompleted) return;

        // retrieve device restore_state
        const device_restore_state = device.settings.hasOwnProperty("restore_state") ? device.settings.restore_state : {};

        // merge into single restore_state
        const restore_state = { ...this.shared_restore_state, ...device_restore_state };

        // apply state (if we have any)
        if (Object.keys(restore_state).length == 0) return;
        this.logger.debug(`Applying initial state '${JSON.stringify(restore_state)}' to '${device.name}' (${device.ieeeAddr})`);
        this.mqtt.publish(
            `${device.name}/set`,
            JSON.stringify(restore_state),
            undefined,
            undefined,
            false,
            false,
        );
    }
}

module.exports = Restore;

I dove right in with the goods, I have had this one or a while and use it mostly for my Ubisys S1/S2, although they do support StartUpOnOff of the genOnOff cluster the implementation on their end is broken and it always results in a toggle action regardless of what you write to it. They are aware of the issue, but it looks unlikely they will fix it.

Hence the need for this extensions, it's pretty simple it attaches to the deviceAnnounce event and then simply restores the state it has read from the configuration file. It could probably be improved a bit by not using this.mqtt.emit() but directly calling the converters, however this works too and keeps the extension simple.

Hopefully this is useful for someone.

[sjorge]

@Koenkk I noticed this stopped working on the latest dev, could it be the refactor to use TS broke all legacy extensions?

onZigbeeEvent() never seems to fire.

[sjorge]

I see, whats the proper way to import the Extension class? Although I was able to attach the onDeviceAnnounce function to the envent bus in start() without it... however I'm not sure how to process the data object that is passed.

I managed to get interviewCompleted via data.device.zh.interviewCompleted but I was unable to get to the device specific settings like before via resolvedEntity.settings as the data passed on the onDeviceAnnounce event seems limited.

[sjorge]

I also notice that when you edit and hit save in the frontend, the extension gets loaded and stuff does get execute multiple tiles and z2m throws an exception for each time save was used in the frontend... althoug that might be due to my new and possible incorrect code.

e.g.

    at Restore.stop (/opt/zigbee2mqtt/dist/util/externally-loaded.js:79:23)
    at Controller.callExtensions (/opt/zigbee2mqtt/lib/controller.ts:312:40)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Controller.stop (/opt/zigbee2mqtt/lib/controller.ts:192:9)
    at stop (/opt/zigbee2mqtt/index.js:101:5)
    at process.handleQuit (/opt/zigbee2mqtt/index.js:107:9))
Zigbee2MQTT:error 2021-10-15 15:03:29: Failed to call 'Restore' 'stop' (TypeError: this.eventBus.removeListenersExtension is not a function
    at Restore.stop (/opt/zigbee2mqtt/dist/util/externally-loaded.js:79:23)
    at Controller.callExtensions (/opt/zigbee2mqtt/lib/controller.ts:312:40)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Controller.stop (/opt/zigbee2mqtt/lib/controller.ts:192:9)
    at stop (/opt/zigbee2mqtt/index.js:101:5)
    at process.handleQuit (/opt/zigbee2mqtt/index.js:107:9))

Currently t he reworked code I have is, which is still not working obviously.

// REMOVE CAUSE BROKEN AF CODE

On closer inspection it looks like z2m initialize the original extension for every time save is used, and not the new one on top of the previous one.

[sjorge]

I was looking at the module/device.ts but I really can't figure out how to get the device specific settings, I managed to find interviewComplete after digging into zh first.

[sjorge]

Made some progress, seeing more weird things will hack on it some more to see if they are my fault or something else is amis.

[Koenkk]

Getting the setting is much more easy now, each event e.g. device interview will have a property device. From this device you can retrieve the settings like device.settings or e.g. name with device.name (https://github.com/Koenkk/zigbee2mqtt/blob/master/lib/model/device.ts)

[okastl]

Sjorge, thank you for you extension, it has been very useful in the past!
Just some suggestions:
With the current dev branch you don't need to merge shared_restore_state and device_restore_state, as the values in device.settings are already merged.
So you can strip down the code in start() and onDeviceAnnounce().
And const device = this.zigbee.resolveDevice(data.device.ieeeAddr); made me smile. You could just write const device = data.device;

Unfortunately I found no solution for TypeError: this.eventBus.removeListenersExtension is not a function.
Is this a z2m bug or needs something to be changed in the extension stop() function?

[sjorge]

And const device = this.zigbee.resolveDevice(data.device.ieeeAddr); made me smile. You could just write const device = data.device;

When I was trying to fix this, the result between the two was not the same.

With the current dev branch you don't need to merge shared_restore_state and device_restore_state, as the values in device.settings are already merged.

Yeah there is definitely room for improvement but there is currently still a bug that requires a full restart because when trying to load it via mqtt (e.g. the frontend) it starts multiple instances, my network israther big and after ever restart it's rather on stable so i didn't really spend the time to poke this much futher.

[sjorge]

I just did some testing, looking up the shared state is still needed:

device={"color_temp":250}, merged={"state":"ON","color_temp":250}

Is logged when I add

this.logger.error(`device=${JSON.stringify(device_restore_state)}, merged=${JSON.stringify(restore_state)}`);

below the cons of restore_state

[okastl]

Yeah there is definitely room for improvement but there is currently still a bug that requires a full restart because when trying to load it via mqtt (e.g. the frontend) it starts multiple instances, my network israther big and after ever restart it's rather on stable so i didn't really spend the time to poke this much futher.

I observe the same problem with two extensions I have written. I assume it is because
async stop() { this.eventBus.removeListeners(this.constructor.name); }
throws an exception TypeError: this.eventBus.removeListenersExtension is not a function.
The question is, is the extension supposed to do something different, or is this a z2m bug?

[christoph-ng]

Does anaybody know if this actually works?

[sjorge]

It used to work but I never updated when the extension format changed.

I mostly now just enforce this using node-red and triggering this based on the events emitted over mqtt. The speed was about the same and it's more stable to do it from node-red.

[okastl]

I have updated it for the new format, but haven't used it since then. Shall I upload it here?

[sjorge]

Sure! It will help people that come here and want to try it out.

[okastl]

Updated extension. No guarantee, use at your own risk...

/*
 * This extension will restore a predefined state to a device on deviceAnnouncement
 *
 * In `configuration.yaml` under the device add a `restore_state` object, per-device
 * settings will overwrite device_options if both have the same property.
 *
 * device_options:
 *   debounce: 1
 *     restore_state:
 *       state: OFF
 * devices:
 *   '0x000d6ffffe8e8d4f':
 *     friendly_name: masterbedroom/lamp
 *     restore_state:
 *       state: ON
 *       color: "#FF0000"
 *
 */
 
class Restore {
    constructor(zigbee, mqtt, state, publishEntityState, eventBus, settings, logger) {
        this.zigbee = zigbee;
        this.mqtt = mqtt;
        this.state = state;
        this.eventBus = eventBus;
        this.settings = settings;
        this.logger = logger;
    }

    async start() {
        this.logger.info('Starting device state restore');

        // attach to events
        this.eventBus.on('deviceAnnounce', this.onDeviceAnnounce.bind(this), this);
    }

    async stop() {
        this.eventBus.removeListeners(this);
    }

    async onDeviceAnnounce(data) {
        const device = data.device;
        
        // abort early if we are interviewing
        if (!device.zh.interviewCompleted) return;

        // retrieve device restore_state
        var settings = device.options;
            
        if( settings === undefined ) {
            settings = device.settings;
        }
        
        const restore_state = settings.hasOwnProperty("restore_state") ? settings.restore_state : {};

        // apply state (if we have any)
        if (Object.keys(restore_state).length == 0) return;
        this.logger.debug(`Applying initial state '${JSON.stringify(restore_state)}' to '${device.name}' (${device.ieeeAddr})`);
        this.mqtt.publish(
            `${device.name}/set`,
            JSON.stringify(restore_state),
            undefined,
            undefined,
            false,
            false,
        );
    }
}

module.exports = Restore;

[okastl]

Note, this here was a compatibility kludge, so the extension could work with older & newer z2m versions:

       // retrieve device restore_state
        var settings = device.options;
            
        if( settings === undefined ) {
            settings = device.settings;
        }

Unfortunately I forgot which is which.
IIRC device.options is the new name, so the if( settings === undefined ) ... block can be removed.

[christoph-ng]

It doesnt work for me. Is it possible to get an example of your node red flow?

[sjorge]

Sure, although mine is part of a complex subflow, I made a short example (that I tested and is working)

[
    {
        "id": "53d335b5814e0efa",
        "type": "mqtt in",
        "z": "ac39b2a656833078",
        "name": "",
        "topic": "zigbee2mqtt/bridge/event",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "ece42dd1.1e82e",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 270,
        "y": 120,
        "wires": [
            [
                "c5e8bcfd03ec2ded"
            ]
        ]
    },
    {
        "id": "c5e8bcfd03ec2ded",
        "type": "function",
        "z": "ac39b2a656833078",
        "name": "restore device state",
        "func": "if (msg.payload.type && msg.payload.type === \"device_announce\") {\n    switch (msg.payload.data.friendly_name) {\n        case \"light/office/desk/bulb\":\n            return {\n                \"topic\": `zigbee2mqtt/${msg.payload.data.friendly_name}/set`,\n                \"payload\": {\n                    \"state\": \"ON\",\n                    \"brightness\": 128,\n                },\n            };\n        case \"light/office/main/panel1\":\n        case \"light/office/main/panel2\":\n        case \"light/office/main/panel3\":\n        case \"light/office/main/panel4\":\n            return {\n                \"topic\": `zigbee2mqtt/${msg.payload.data.friendly_name}/set`,\n                \"payload\": {\n                    \"state\": \"OFF\",\n                },\n            };\n        default:\n            // NOTE: do nothing if unknown friendly_name\n            break;\n    }\n}\nreturn null;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 530,
        "y": 120,
        "wires": [
            [
                "a3d9b402f53cb847",
                "083b545f29c1eb33"
            ]
        ]
    },
    {
        "id": "a3d9b402f53cb847",
        "type": "mqtt out",
        "z": "ac39b2a656833078",
        "name": "",
        "topic": "",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "ece42dd1.1e82e",
        "x": 750,
        "y": 120,
        "wires": []
    },
    {
        "id": "083b545f29c1eb33",
        "type": "debug",
        "z": "ac39b2a656833078",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 760,
        "y": 200,
        "wires": []
    },
    {
        "id": "ece42dd1.1e82e",
        "type": "mqtt-broker",
        "name": "mosquitto",
        "broker": "localhost",
        "port": "1883",
        "clientid": "nodered",
        "usetls": false,
        "compatmode": false,
        "protocolVersion": "5",
        "keepalive": "60",
        "cleansession": false,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "",
        "willQos": "0",
        "willPayload": "",
        "willMsg": {},
        "sessionExpiry": ""
    }
]

So instead of having the state to restore in configure.yaml, you have it in the function node.

[christoph-ng]

Works create, thank you.

[taylordtriplett]

Sure, although mine is part of a complex subflow, I made a short example (that I tested and is working)

[
    {
        "id": "53d335b5814e0efa",
        "type": "mqtt in",
        "z": "ac39b2a656833078",
        "name": "",
        "topic": "zigbee2mqtt/bridge/event",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "ece42dd1.1e82e",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 270,
        "y": 120,
        "wires": [
            [
                "c5e8bcfd03ec2ded"
            ]
        ]
    },
    {
        "id": "c5e8bcfd03ec2ded",
        "type": "function",
        "z": "ac39b2a656833078",
        "name": "restore device state",
        "func": "if (msg.payload.type && msg.payload.type === \"device_announce\") {\n    switch (msg.payload.data.friendly_name) {\n        case \"light/office/desk/bulb\":\n            return {\n                \"topic\": `zigbee2mqtt/${msg.payload.data.friendly_name}/set`,\n                \"payload\": {\n                    \"state\": \"ON\",\n                    \"brightness\": 128,\n                },\n            };\n        case \"light/office/main/panel1\":\n        case \"light/office/main/panel2\":\n        case \"light/office/main/panel3\":\n        case \"light/office/main/panel4\":\n            return {\n                \"topic\": `zigbee2mqtt/${msg.payload.data.friendly_name}/set`,\n                \"payload\": {\n                    \"state\": \"OFF\",\n                },\n            };\n        default:\n            // NOTE: do nothing if unknown friendly_name\n            break;\n    }\n}\nreturn null;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 530,
        "y": 120,
        "wires": [
            [
                "a3d9b402f53cb847",
                "083b545f29c1eb33"
            ]
        ]
    },
    {
        "id": "a3d9b402f53cb847",
        "type": "mqtt out",
        "z": "ac39b2a656833078",
        "name": "",
        "topic": "",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "ece42dd1.1e82e",
        "x": 750,
        "y": 120,
        "wires": []
    },
    {
        "id": "083b545f29c1eb33",
        "type": "debug",
        "z": "ac39b2a656833078",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 760,
        "y": 200,
        "wires": []
    },
    {
        "id": "ece42dd1.1e82e",
        "type": "mqtt-broker",
        "name": "mosquitto",
        "broker": "localhost",
        "port": "1883",
        "clientid": "nodered",
        "usetls": false,
        "compatmode": false,
        "protocolVersion": "5",
        "keepalive": "60",
        "cleansession": false,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "",
        "willQos": "0",
        "willPayload": "",
        "willMsg": {},
        "sessionExpiry": ""
    }
]

So instead of having the state to restore in configure.yaml, you have it in the function node.

I have an extremely nube question, so sorry in advance. The devices(in the function node) are typed as -
case "light/office/main/panel1":

What is that formatting in relation to my device names? The friendly device name in Z2MQTT is "Upstairs Hallway Light 1". I also have this in an HA group named "Upstairs Hallway Lights". Just trying to figure out how to type that in your msg. Thanks!

[sjorge]

It's just the friendly name, mine happen to contain / to make subscribing to certain subsets easier

[martintilma]

Today I found this discussion in my search on how to set the light bulb when it was switched on by the physical switch. Inspired by the comment on using node-red I tried to do it with an automation in Home Assistant.

Here is my result:

alias: Keuken Tafellampen power on
description: ""
trigger:
  - platform: mqtt
    topic: zigbee2mqtt/bridge/event
    payload: >-
      {"data":{"friendly_name":"Keuken tafel lamp
      1","ieee_address":"0xf082c0fffe3e30ca"},"type":"device_announce"}
condition: []
action:
  - service: light.turn_on
    data:
      color_temp: 330
      brightness_pct: 75
    target:
      entity_id: light.keuken_tafel_lampen
mode: single

(identifiers are in Dutch)

It might help others