Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend service to suggest addons via generic IP scan #3936

Open
4 of 11 tasks
holgerfriedrich opened this issue Dec 19, 2023 · 18 comments
Open
4 of 11 tasks

Extend service to suggest addons via generic IP scan #3936

holgerfriedrich opened this issue Dec 19, 2023 · 18 comments
Labels
enhancement An enhancement or new feature of the Core

Comments

@holgerfriedrich
Copy link
Member

holgerfriedrich commented Dec 19, 2023

PR #3920 introduced a (partly) generic addon finder, which allows to discover devices by sending single IP frames. This is useful for devices which do not actively announce and require a search request to a multicast address. Maybe the word scanning is a bit misleading, at the moment it is just a single search frame, and only once at start-up in the current implementation.

Note: if case you can use mdns or upnp, there are separate addon finders for this! If you want to detect a service running locally, there is one for processes as well.

The implementation is rather tuned to a single use right now - detecting KNX installations - but is intended to be extended towards release 4.2. This issue is to collect requirements from other addons which can be used when starting a generalization.

The configuration is via addon.xml and looks like this (see #3948 for developer documentation):

	<discovery-method>
		<discovery-method>
			<service-type>ip</service-type>
			<discovery-parameters>
				<discovery-parameter>
					<name>type</name>
					<value>ipMulticast</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>destIp</name>
					<value>224.0.23.12</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>destPort</name>
					<value>3671</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>request</name>
					<value>0x06 0x10 0x02 0x01 0x00 0x0e 0x08 0x01 $srcIp $srcPort</value>
				</discovery-parameter>
				<discovery-parameter>
					<name>timeoutMs</name>
					<value>5000</value>
				</discovery-parameter>
			</discovery-parameters>
			<match-properties>
				<match-property>
					<name>response</name>
					<regex>.*</regex>
				</match-property>
			</match-properties>
		</discovery-method>
	</discovery-methods>

Frames to be sent are specified in the parameter section, and dynamic replacement of source IP and port is possible. Any return frame is matched, only the catch all .* is supported.

addon detection method scan interval dynamic data parsed response comment
knx mulitcast frame (hex data) once srcIp, srcPort .* #3912 (merged)
ipcamera multicast frame (xml) tbd randomNumber .* #3943 openhab/openhab-addons#16090 (merged)
govee multicast frame tbd no .* requires listening on port 4002, openhab/openhab-addons#16109 (merged)
zway unicast, scan local network tbd none http code 200 see below
danfoss broadcast tbd none static content #4036 (merged)

(table to be continued)

Features to be developed (incomplete list):

  • plaintext request instead of hex bytes ✅
  • send at regular intervals (add-on specific setting)
  • real regex matching for result frames (match packet as hex stream, and loop over all result frames until match is found or timeout occurs)
  • new dynamic data, e.g. randomNumber of certain length (✅ for uuid sting)
  • possibly caching to avoid sending the same query twice
  • skip detection if binding is already installed ✅
  • unicast
  • broadcast ✅
  • parallel sending (avoid that suggestion is missed if the checker was not running before wizard starts - UI could also be improved to wait until all checks finished) [WIP] IP Add-on Finder: make scans asynchronously #4094
  • respecting network config from Add network-interface context #3981
  • cleaning up the log levels (debug for status messages, trace for data on the network)
@holgerfriedrich holgerfriedrich added the enhancement An enhancement or new feature of the Core label Dec 19, 2023
@openhab-bot
Copy link
Collaborator

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-get-an-addon-to-show-up-in-the-new-addon-suggestion-finder-list/152127/13

@Skinah
Copy link
Contributor

Skinah commented Dec 20, 2023

To summarize what is needed for the ipcamera addon as discussed in the forum link:

To discover ONVIF cameras it uses port 3702 and IP multicast address 239.255.255.250 using what is called WS-Discovery. It is just the following SOAP/XML packet sent.

<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:randomUUID()</w:MessageID><w:To e:mustUnderstand="true">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To><w:Action a:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action></e:Header><e:Body><d:Probe><d:Types xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dp0="http://www.onvif.org/ver10/network/wsdl">dp0:NetworkVideoTransmitter</d:Types></d:Probe></e:Body></e:Envelope>

Challenges identified are:

  1. In the above XML packet you need to use the function java.util.UUID.randomUUID() to replace the text randomUUID() It needs to be different for each scan.
  2. How to enter in XML contents into the addon.xml file without causing issues.
  3. What about when a reply could match multiple bindings, do we do the scan multiple times and then use a different filter, or can they be combined? Doorbird is a binding for that brand of ipcamera which can also do ONVIF, so it would match two bindings. Do we scan once and then filter the cached results differently in each binding, or do the whole scan multiple times?

Good news is that because the XML states NetworkVideoTransmitter in the request, all replies will only be from devices that the binding can talk to. No filtering needed on replies since this is just for just putting the binding forward to be installed where further logic can be used.

@mherwege
Copy link
Contributor

3. What about when a reply could match multiple bindings, do we do the scan multiple times and then use a different filter, or can they be combined? Doorbird is a binding for that brand of ipcamera which can also do ONVIF, so it would match two bindings. Do we scan once and then filter the cached results differently in each binding, or do the whole scan multiple times?

If both bindings define the same discovery criteria, they will both be suggested. It is for the user to decide what to install. No 2 scans are required for this.

@holgerfriedrich
Copy link
Member Author

holgerfriedrich commented Dec 20, 2023

@Skinah Would the following string work? (this is the result of a draft implementation)
Is UUID fine?

<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:1534bd9d-f9c1-45cc-a747-311d6a76db3b<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:)</w:MessageID><w:To e:mustUnderstand="true">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To><w:Action a:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action></e:Header><e:Body><d:Probe><d:Types xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dp0="http://www.onvif.org/ver10/network/wsdl">dp0:NetworkVideoTransmitter</d:Types></d:Probe></e:Body></e:Envelope>

@holgerfriedrich
Copy link
Member Author

holgerfriedrich commented Dec 23, 2023

@stefan-hoehn Mechanism for Govee
#3920 (comment)
can be discussed here.

Developer docs are not yet merged, see #3948.

request does only support hex bytes by now, so you could make it work if you use
<value>0x7B 0x22 0x6D 0x73 0x67 0x22 0x3A 0x20 0x7B 0x22 0x63 0x6D 0x64 0x22 0x3A 0x20 0x22 0x73 0x63 0x61 0x6E 0x22 0x2C 0x20 0x22 0x64 0x61 0x74 0x61 0x22 0x3A 0x20 0x7B 0x22 0x61 0x63 0x63 0x6F 0x75 0x6E 0x74 0x5F 0x74 0x6F 0x70 0x69 0x63 0x22 0x3A 0x20 0x22 0x72 0x65 0x73 0x65 0x72 0x76 0x65 0x22 0x7D 0x7D 0x7D</value>

Not sure about port 4002.... do we have to listen on 4002 or is if source port of the response?

@holgerfriedrich
Copy link
Member Author

@stefan-hoehn just updated the comment above to include an example....

@stefan-hoehn
Copy link
Contributor

stefan-hoehn commented Dec 23, 2023

This is how it works:

  • send a multicast request to port 4001
  • and the clients send and asynchronous response back on port 4002, so we need to listen explicitly on 4002
  • the response then is as follows
{
   "msg":{
      "cmd":"scan",
      "data":{
        "ip":"192.168.1.23",
        "device":"1F:80:C5:32:32:36:72:4E",
        "sku":"Hxxxx",
        "bleVersionHard":"3.01.01",
        "bleVersionSoft":"1.03.01",
        "wifiVersionHard":"1.00.10",
        "wifiVersionSoft":"1.02.03"
      }
    }
  }
 
image

But I think noticing that we get ANY response would be enough, I would say.

@mherwege
Copy link
Contributor

@holgerfriedrich Looking at this, and the ipcamera request, I think an alternative (non-hex) request should be supported.

@holgerfriedrich
Copy link
Member Author

@holgerfriedrich Looking at this, and the ipcamera request, I think an alternative (non-hex) request should be supported.

:-)

Looks Like.... But we need to make sure to escape tags - otherwise the XML parser might do something wrong....

@mherwege
Copy link
Contributor

mherwege commented Dec 24, 2023

@mherwege I don’t think that is the responsibility of the code. When defining the discovery criteria in the xml, it is sufficient to properly escape the special characters: https://stackoverflow.com/questions/1091945/what-characters-do-i-need-to-escape-in-xml-documents
But that would be true for any of the finders.

@holgerfriedrich
Copy link
Member Author

@mherwege Yes, lets go for a requestPlain which uses org.openhab.core.util.StringUtils.StringUtilsunEscapeXml()..
Developers of bindings need to escape the characters mentioned here by hand.
Better than using the hex editor to transform the query into something unreadable.

@stefan-hoehn
Copy link
Contributor

I can confirm that the implementation works well already for the Govee-Binding. See #16109

@holgerfriedrich
Copy link
Member Author

andrewfg commented Dec 27, 2023

@holgerfriedrich the ZWay binding is probably a candidate to be suggested via your IP addon finder. => WDYT?

https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.binding.zway#discovery

This is about the zway gateways connected to the LAN.

The documentation of the zway binding states the following:

Z-Way doesn't support any discovery protocol like UPnP for this purpose. That is why first all IP addresses in local network are checked on port 8083. If the server answers, a ZAutomation request (/ZAutomation/api/v1/status) is performed to ensure, the found server runs Z-Way.

This would be the first real "IP scan", i.e. iterating over all local IP addresses finding a gateway.

It would require a few new features:

  • new type "unicast"
  • allowing a dynamic IP range (e.g. local subnet of each interface) - but still limiting the number of scanned hosts to avoid the problems we have seen with Docker
  • parsing the response (likely we need to match a http status code)
  • and because this is creating some load, I would also suggest that we do this only if the binding is not installed

Did I miss something, @andrewfg ?

@andrewfg
Copy link
Contributor

Did I miss something

I think it pings all port 8083 on the subnet, and if the port is open it does a specific HTTP GET for a URL on that host:8083, and if it returns HTTP 200 it would become a suggestion.

@holgerfriedrich
Copy link
Member Author

I think it pings all port 8083 on the subnet, and if the port is open it does a specific HTTP GET for a URL on that host:8083, and if it returns HTTP 200 it would become a suggestion.

Yes, this sounds reasonable, thanks.

A subnet could be quite large, I would limit it a /24 and exclude the broadcast address (such that we have to scan 255 IPs per interface at max).

@mherwege
Copy link
Contributor

I don’t think we should allow scans on all interface. See discussion here, how it can cause drama with Docker: https://community.openhab.org/t/unsuccess-openhab-updated-from-4-0-2-to-4-1-0-on-docker-openhab-unresponsive/152282/24

OH does have a setting for the primary IP. Could we restrict it to thecsubnet of that iP?

I even think we may want to limit the current IP finder implementation to only use one interface (or make that configurable).

@mlobstein
Copy link
Contributor

If regex matching is implemented then the epsonprojector and sonyprojector addons could be detected as many newer models respond to Control4 SDDP discovery probes as described here:

https://github.com/sammck/sddp-discovery-protocol/blob/main/README.md

SDDP is similar to UPNP/SSDP in that a multicast search frame is sent and any supported devices respond back to the multicast address with a packet containing device information. By using a regex these packets would be filtered to determine the device type and manufacturer.

@andrewfg
Copy link
Contributor

@mlobstein I created #4234 for this..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An enhancement or new feature of the Core
Projects
None yet
Development

No branches or pull requests

7 participants