Service to suggest addons via generic IP scan

[holgerfriedrich]

This is the implementation of a generic IP-based add-on suggestion finder.
It will try to discover add-ons by scanning the IP network with specific packages.

The description is taken from the metadata.

At startup, a single scan is done in a separate thread.

The finder can be disabled, as it actively sends packets to the network which might not be desired.

Example configuration:
for KNX we want to send a multicast package to 224.0.23.12 port 3671.

The section in addon.xml looks like this:

		<discovery-method>
			<service-type>ip</service-type>
			<match-properties>
				<match-property>
					<name>type</name>
					<regex>ip_multicast</regex>
				</match-property>
				<match-property>
					<name>dest_ip</name>
					<regex>224.0.23.12</regex>
				</match-property>
				<match-property>
					<name>dest_port</name>
					<regex>3671</regex>
				</match-property>
				<match-property>
					<name>request</name>
					<regex>0x06 0x10 0x02 0x01 0x00 0x0e 0x08 0x01 $src_ip $src_port</regex>
				</match-property>
				<match-property>
					<name>timeout_ms</name>
					<regex>5000</regex>
				</match-property>
				<match-property>
					<name>response</name>
					<regex>any</regex>
				</match-property>
			</match-properties>
		</discovery-method>

Refs: #3912

[mherwege]

> <discovery-method>
> 			<service-type>ip</service-type>
> 			<match-properties>
> 				<match-property>
> 					<name>type</name>
> 					<regex>ip_multicast</regex>
> 				</match-property>
> 				<match-property>
> 					<name>dest_ip</name>
> 					<regex>224.0.23.12</regex>
> 				</match-property>
> 				<match-property>
> 					<name>dest_port</name>
> 					<regex>3671</regex>
> 				</match-property>
> 				<match-property>
> 					<name>request</name>
> 					<regex>0x06 0x10 0x02 0x01 0x00 0x0e 0x08 0x01 $src_ip $src_port</regex>
> 				</match-property>
> 				<match-property>
> 					<name>timeout_ms</name>
> 					<regex>5000</regex>
> 				</match-property>
> 				<match-property>
> 					<name>response</name>
> 					<regex>any</regex>
> 				</match-property>
> 			</match-properties>
> 		</discovery-method>

It feels to me like we are trying to shoe-horn the parameters for this into the existing schema. You may have noticed that the mdns discovery has an extra parameter in the schema. I would suggest to extend the schema for this.
Also, I don't like using a field called regex and filling it with something else.

I think you should keep the check for the return value to be a pure regex (you can express hex numbers in a regex as well). All the others are not matching criteria, but input parameters.

@andrewfg What do you think? In hindsight, we should probably have defined the mdnsServiceType parameter in the schema differently to make it more generalizable, something like:

<discovery-method>
 	<service-type>mdns</service-type>
 	<parameters>
 	 	<parameter>
 	 	 	<name>mdnsServiceType</name>
 	 	 	<value>...</value>
 	 	</parameter>
  	</parameters>
 	<match-properties>
		<match-property>
 			<name>...</name>
 			<regex>...</regex>
 		</match-property>
 	</match-properties>
</discovery-method>

That would be extendable without modifying the schema.

[mherwege]

At startup, a single scan is done in a separate thread.

Shouldn't we run this on a schedule and repeat it at regular intervals? The interval could be another configuration (next to switching this finder on/off completely).

[holgerfriedrich]

Thanks, Mark, for looking into this.

I wanted to see it working first, changing the schema is probably something to consider, as the regex for input seems wrong to me as well.

I am struggling a bit putting the scan from the getter function to task scheduled from the ctor... The list of candidates is now empty. I need to check later.

[andrewfg]

@holgerfriedrich yup, the way that you mis- use the schema is definitely a horrible hack; but as a proof of concept this approach makes sense; however I still wonder if there are other add-ons other than KNX where such a network probe is useful. ??

[andrewfg]

That would be extendable without modifying the schema.

@mherwege we would still have to modify (extend) the schema to support the new parameter(s) elements. But I get what you are saying about the extensibility. But we should wait until after v4.1 for this. Let is see whether other finders require a yet more sophisticated model. Note my draft for the USB device related bindings #3922 would not need extra parameters.

[holgerfriedrich]

Proof of concept is working,

[holgerfriedrich]

At startup, a single scan is done in a separate thread.

Shouldn't we run this on a schedule and repeat it at regular intervals? The interval could be another configuration (next to switching this finder on/off completely).

KNX is not something you turn on and off, so I think it is fine to trigger it once at startup.
Likely this is desired for other bindings.
We need to analyze first which devices we want to detect using this approach.

[holgerfriedrich]

we should probably have defined the mdnsServiceType parameter in the schema differently to make it more generalizable, something like:

<discovery-method>
 	<service-type>mdns</service-type>
 	<parameters>
 	 	<parameter>
 	 	 	<name>mdnsServiceType</name>
 	 	 	<value>...</value>

This is probably a good idea.... Should we change it before this gets adopted in more places like the marketplace?

[mherwege]

we should probably have defined the mdnsServiceType parameter in the schema differently to make it more generalizable, something like:

<discovery-method>
 	<service-type>mdns</service-type>
 	<parameters>
 	 	<parameter>
 	 	 	<name>mdnsServiceType</name>
 	 	 	<value>...</value>

This is probably a good idea.... Should we change it before this gets adopted in more places like the marketplace?

That is also my thought. Better change it now before release than having to change it later and have a larger impact.
It still requires quite some changes though:

• Changes in core to AddonInfo and the mDNS finder
• Update to the schema and upload the new schema
• PR in add-ons repo to change all mDNS discovery methods

If we wait until after release, we could keep the existing mdnsServiceType for backward compatibility in the code but encourage the use of the parameter syntax.

@andrewfg @kaikreuzer What do you think? This is not an immediate need, but will avoid some hassle in the future? Is there still time to do this change?

[J-N-K]

Please do those changes before the release. In general I think this should also increase the version, but @kaikreuzer can probably comment on why we haven't done so in the past.

[mherwege]

@andrewfg I will create a PR to adapt core.

[andrewfg]

@mherwege Ok I will check your core PR

I will also do a PR for the developer docs

[mherwege]

Addon PR also created, changing all mDNS discovery definitions in the addon.xml files: openhab/openhab-addons#16060

[holgerfriedrich]

Adapted to #3924, will not work without

[kaikreuzer]

In general I think this should also increase the version, but @kaikreuzer can probably comment on why we haven't done so in the past.

Afair, we did backward compatible changes in the past and therefore kept the version, so that there wasn't any need to update the xmls in the add-ons (and therefore keep e.g. the ones in the marketplace compatible). With breaking changes, we should probably indeed increase the version. But for #3924, it is imho ok to stick to the version as there hasn't been a release yet.

[holgerfriedrich]

@mherwege @andrewfg thanks a lot for your work on generalizing the schema!

I have rebased to current master which contains the new schema.

@mherwege @kaikreuzer
This is now more or less at the level of a proof of concept. However, I would very much appreciate if I could get it into 4.1 to detect KNX installations. There has been more activity to make the whole chain working: discovery service to detect interfaces in the binding itself, and process suggestion finder which can already identify a running knxd deamon. The only piece missing is this one.
I know that this somehow needs to be balanced with the approach of only putting mature and already generalized features into core.

Can you give advice what are the most important pieces missing right now?

• How do I care for the activation/deactivation? I just extended addons.xml in org.openhab.core and added the i18n strings until now.
• Do you think it is important to clean the pattern matching for received packets? Right now it works only with the first received frame and the pattern ".*".

@kaikreuzer What it the time frame this needs to be done? Are we too late for 4.1 already?

[kaikreuzer]

@holgerfriedrich I am not a big fan of IP scans and all sorts of side-effects might occur here. I'd therefore prefer to not merge this as a PoC in the very last minute, but rather test it thoroughly after the 4.1 release is done.

[holgerfriedrich]

@holgerfriedrich I am not a big fan of IP scans and all sorts of side-effects might occur here. I'd therefore prefer to not merge this as a PoC in the very last minute, but rather test it thoroughly after the 4.1 release is done.

I get you point.... Though, IP scan sounds a bit more aggressive than it looks to me in this case: A single UDP frame to a well defined multicast addess and port. One frame sent during startup. And we can even disable it.
We are going a big step towards better experience to new users - just wanted to be part of this with KNX as well.

[holgerfriedrich]

Just a note, I was not able to deactivate any of the finders. I deactivated via UI, restarted, and everything was discovered and shown in the wizard (avm, etc ) - what did I wrong?

Did you wait long enough before restarting? It can take 1 minute for the change to be effective.

ok, confirmed, did the restart too early.... it works when I wait longer.

[mherwege]

LGTM with refinements to be done for other IP finder scenarios. But this should not stop making this availalble already.

[holgerfriedrich]

LGTM with refinements to be done for other IP finder scenarios. But this should not stop making this availalble already.

@kaikreuzer Are we too late? Did you already trigger the milestone build?
Please also merge openhab/openhab-addons#16055, thanks!

[kaikreuzer]

Thanks - just some logging improvement suggestions, then we are good to merge.

[holgerfriedrich]

Thanks - just some logging improvement suggestions, then we are good to merge.

done, thanks

[kaikreuzer]

Thanks!

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-get-an-addon-to-show-up-in-the-new-addon-suggestion-finder-list/152127/6

[stefan-hoehn]

@mherwege I looked into this because currently the Govee binding isn't detected. After doing some research these are the ways to find out if a Govee device is on the network

• by pinging "ping -c 1 239.255.255.250"
• or by sending a "{"msg": {"cmd": "scan", "data": {"account_topic": "reserve"}}}" via Multicast ip 239.255.255.250:4001 and receiving back an answer on our port: 4002 (like the binding does).

Do you see any way based on this implementation to auto discover the binding ?

e.g.

``
Note that request is on 4001 but the answer would be on 4002..

<discovery-method>
			<service-type>ip</service-type>
			<discovery-parameters>
				<discovery-parameter>
					<name>type</name>
					<value>ipMulticast</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>destIp</name>
					<value>239.255.255.250</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>destPort</name>
					<value>4001</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>request</name>
					<value>{"msg": {"cmd": "scan", "data": {"account_topic": "reserve"}}}</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>timeoutMs</name>
					<value>5000</value>
				</discovery-parameter>
			</discovery-parameters>
			<match-properties>
				<match-property>
                                         <!-- not sure what to put in here ?? -->
					<name>response</name>
					<regex>.*</regex>
				</match-property>
			</match-properties>
		</discovery-method>
	</discovery-methods>

[mherwege]

Not at this point, but see #3936. We are looking to document different cases so we can make sure a more complete solution gets implementeren for 4.2.

[stefan-hoehn]

Thanks, let me know when I can try out something for #3936 or support you there.