Pico W: Issues with ussl / usocket with encrypted MQTT

[IanW6374]

Hi - I am seeing error "MQTTException: 5" when trying to connect to an authenticated HiveMQ MQTT broker when connecting via TLS.

To try and narrow down the issue I have tested just sending the same raw data for a MQTT CONNECT message from both MicroPython and Python.

Server Response:-

MicroPython: \x20\x00\x05 e.g. "MQTTException: 5"
Python: \x20\x00\x00 e.g. "MQTT CONNECT ACK"

As we are sending the same data to the server then it would seem like an issue with the ussl / usocket module ???

The rawdata test code is shown below (Note: the server and msg have been truncated as it has password information)

`
ca_certificate_path = "/certs/catrust-micropython.der"
print('Loading CA Certificate')
with open(ca_certificate_path, 'rb') as f:
cacert = f.read()
print('Obtained CA Certificate')

server = '206..............5571eac9.s1.eu.hivemq.cloud'
port = 8883
ssl_params = {'server_side':False, 'key':None, 'cert':None, 'cadata':cacert, 'cert_reqs':ssl.CERT_REQUIRED}

msg = b'\x10\x2a\x00\x04\x4d\x51.......................\x6d\x73\x59\x78'

sock = socket.socket()
addr = socket.getaddrinfo(server, port)[0][-1]
sock.connect(addr)
sock = ssl.wrap_socket(sock, **ssl_params)

sock.write(msg)

resp = sock.read(4)

print('Response: ',resp)

`

Any thoughts ???

[Carglglz]

@IanW6374 could you add this to verify that sock.write(msg) is sending the whole message?

...
print(len(msg))
print(sock.write(msg))
...

From MicroPython docs about socket.write

Write the buffer of bytes to the socket. This function will try to write all data to a socket (no “short writes”). This may be not possible with a non-blocking socket though, and returned value will be less than the length of buf.

So both values should be the same, if not then there is the problem.
In case they are the same, I would try to use Wireshark and see if I can spot any differences between Python and MicroPython. 🤔

[IanW6374]

The MQTT CONNECT (msg) is only 44 bytes long and from what I can see the msg (data) is sent within the same packet in both Python and MicroPython

[Carglglz]

@IanW6374 I could reproduce this issue, you need to add server_hostname=server to ssl.wrap_socket, not sure exactly why this happens, but I guess Python does this automatically. 🤷🏼‍♂️ (If your are interested this has to do with SNI)

So the fix should be

ssl_params = {'server_side':False, 'key':None, 'cert':None, 'cadata':cacert, 'cert_reqs':ssl.CERT_REQUIRED, 'server_hostname': server}

[IanW6374]

@Carglglz Thank you for the resolution - Your prompt response is much appreciated !!

[fishjoe]

Ian, can you pls update if it's working now?
Many thanks.

[IanW6374]

@fishjoe It is working, however, I don't get a "mark as answer" but on the part of thread which has the answer.

[fishjoe]

Hi, I have a very noob question hope you guys don't laugh about. Where do you guys get the certificate files from? I'm doing the exactly same thing and got stuck at this ssl thing. When I'm using an iOS app to send and receive MQTT, I only need to "kick in box" as an option labeled "SSL", and everything worked fine. In this case why do I need a seperate file? Many thanks.

[IanW6374]

Hi - You can create the certs with the following openssl command (change "host" and "port" for your requirement)

openssl s_client -connect <host>:<port> -showcerts < /dev/null 2> /dev/null | sed -n '/BEGIN/,/END/p' > host.pem

[fishjoe]

I actually worked out how this would work.
client = MQTTClient(secrets.client_id, secrets.mqtt_server, port=secrets.mqtt_port, user=secrets.mqtt_username, password=secrets.mqtt_psd, ssl=True, ssl_params={ "server_hostname": secrets.mqtt_server})

especially

ssl=True, ssl_params={ "server_hostname": secrets.mqtt_server}

[sgbaird]

@fishjoe does this ensure that your password is encrypted? (i.e. never gets publicly exposed as plain-text)

Related:

• http://www.steves-internet-guide.com/mqtt-username-password-example/
• http://www.steves-internet-guide.com/creating-and-using-client-certificates-with-mqtt-and-mosquitto/
• http://www.steves-internet-guide.com/mosquitto-tls/
• extmod/modussl_axtls: Implement key and cert kw args to wrap_socket. #3398