Introduce energy consumption counters within firmware

[tp1de]

Analysing the energy consumption is a demand from most users. I just implemented this within the ioBroker adapter.

Bosch gateways provide this information (recordings) even historically.
Since I could not recognize any telegrams assosiated with this consumption info, I believe that the recordings are calculated within the Bosch gateway and can be polled by local LAN or cloud API. Recordings consist of hourly, daily, or monthly JSON arrays. for energy consumption or temperatures. In the past 2 years history was available. With last firmware update this was reduced to 12 months. Sample rate is 60 seconds. So hourly values should have 60 samples, daily 1440 . If samples are missing, then the actual values are interpolated within the cloud apps. E.g. 10% samples missing then value is increased by 10%.

I recommend to integrate consumption counters for ww and heating within the ems-esp firmware.
Maybe even separate ones for hybrid systems.

The nominal power for heating and dhw (could be different) must be configuration settings since there are not included yet as entities. Nominal power multiplied with modulation and summed up over an hour should increase new entity energy counters for ch and dhw. Sample rate should be low (10-15 seconds) for accuracy.

The energy counters must be saved on filesystem during reboots / firmware upgrades.

[tp1de]

I screened all telegrams on my heating system (EMS+ with RC310) for telegram values. My nominal power ist 22 KW --> 0x16.
I only find this value in one telegram: switchmode for hc (which I have decoded) ... anyhow, it's not the nominal power.

No other telegram contains 0x16 within data. Or shall I search differently @MichaelDvP?
My assumption is that the nominal power is not within any of telegrams send. (known and unknown ones)

[tp1de]

I have to check how to do ... I do not have the km200 as a separate box but IP-Inside. I have to powerdown the boiler. Since ems-esp is connected to service key and powered there ... I might loose some telegrams during restart.

[MichaelDvP]

Isn't it inside connected with a ems connector (screw-clamps)? Can't you power the ems-esp via usb?

[tp1de]

Yes that's what I have done now. Here ist the start sequence toward the Gateway:
syslog_1693643428347.txt

[MichaelDvP]

Telegram 0x04 could be, offset 4: 16 12 64 => power, min, max modulation. Could that be? My boiler shows 00 64 64, it's unmodulated, so min = 100%.
Can you ask KM200 users with shown nominal power in bosch app to read their telegram 0x04 to check for other boiler? I've already posted to #1148

[tp1de]

Looks good:

[MichaelDvP]

Please specify more detailed what you expect. Not what bosch do in it's cloud, only what is usefull for ems-esp.
We are limited by memory and flash writecycles. The flash have ~100000 write/erase cycles,writing a single cell every 15 sec will kill the flash in 17 days. We have to do wear leveling and enough flashspace to change the cells often (rolling update). But 5760 samples dayly, 2102400 per year does not fit in a few MB flash (if we only save in 16MB systems and skip the 4MB esp32).
I think all statistics with lots of data has to be done on a higher end mashine, reading it from mqtt/api.
The heatpumps provide energy readings for heating/cooling/pool/dhw, ithink they have a integrated energy counter.
The solar modules calculate energy per hour (SM100, ISM) and energy per day (only SM100).
In #1148 there is a discussion about actual power.
One of the problems is, that bosch uses different percentage in the boiler systems, see #908. Sometimes it is percent of nominal power, sometimes of maximum power. Maximum power is sometimes 100% for heating and 130% for dhw, other boilers use 100% for dhw and 65% for heating. To calculate the power we have to know the right way for each boiler. If we add a lot of user defined constants for this , most users do not know what to enter, it's not noted in the manuals.
A possible solution is to calculate every full minute the average modulation for heating and dhw by summing the modulation changes. Then users can read this value and calculate the energy better without polling often.

[tp1de]

@MichaelDvP I do not expect anything .... since I can do the calculation either in ioBroker or HA.
I just thought that it might help most users to have energy counters available for heating and ww.
So no historical values - just persistent counters.

I am not aware about flash cycle times, I just thought that it might help to have this entities already prepared in the ems-esp firmware.
Updates are only needed when energy consumption increases.
Flash cycle time should be then identically to modulation - or? (read every 10 seconds)

And yes the nominal power can be sometimes different for heating and ww preparation. That's why I thought that entering this as absolute values within the settings should be the easiest solution.

Within the ioBroker adapter I already implemented this basic calculation and storing the result within a database.
For HA I have a node-red flow to implement this for the energy dashboard reading the km200 gateway recordings.
This is more complex since I rebuild ha statistics records with sql statements to include historical values.

It could be possible just to build a basic nr flow for creating the energy counters within HA to be used for the energy dashboard.
But the easiest way for HA integration would be to get the energy counters (ch and dhw) with increasing values from ems-esp (... and yes not supporting hybrid systems).
This would avoid a special node-red flow and could be done using existing mqtt discovery.

For end-users this would be the easiest solution.
That's why I started this discussion.

[tp1de]

Sounds good ... mqtt attributes should be: "unit_of_meas":"kWh","state_class":"total_increasing","device_class":"energy",

[proddy]

I would prefer using NVM and the Preferences library (https://www.industrialshields.com/blog/arduino-industrial-1/how-to-use-esp32-nvs-memory-to-store-data-permanently-550). We've talked about this a few times before but can't quickly find the issues.

[tp1de]

Another solution would be reading last value from mqtt on restart ..... needs mqtt broker enabled

[proddy]

that's an interesting concept. EMS-ESP has always been the slave, but it can also be configured to listen to MQTT and API events and provide responses.

[MichaelDvP]

We can publish the energy values retained to a own topic and subscribe to this topic on startup, than the broker will send them. But for api this does not work.

Regarding NVS, we have discussed that for saving the counter values in analogsensors, which is the same task as storing enery values. It has also wear leveling, so we can use it. But it's the same problem with flash endurance, we should store only very few, nessesary values as small as possible and not to often. If the flash in nvs breaks, this does not affect the filesystem, we'll lose only the stored values.

For the energy values we have UBAMonitorFast for the state publish commingaroud every 10 seconds, For not losing resulution we have to calcualte energy in 0.01 Wh (15%, 15kW, 10s => 6.25 Wh) , but the counter could go up very high, In 10 years it could be >500.000 kWh on a powerfull boiler.
Actually ems-esp can not handle longer values as int32 for entity/web interface. So i'll display/publish only full kWh. Internally i'll use double to calculate and store.

[tp1de]

Counter should be saved when a "normal" restart is initiated (firmware upgrade or restart from web-ui). This will not take place too often.
I haven't had crashes since a long time. But a lot of gateways might be connected to service key and powered there.

When such a restart happens then count shall start with 0 if no mqtt broker is enabled.
It's then the responsibility of the api interface (e.g. ioBroker adapter) to recognize this and update the counter. Counter has to be writable.