Issue with reestablishing an SSL connection

[ondrej1024]

I have an issue with reestablishing an MQTT SSL connection after the system has been running for some time.

I can successfully establish an MQTT connection from the client to the broker defining ssl.SSLContext(). When the connection breaks I try to reconnect. This fails in some cases with this traceback:

Traceback (most recent call last):
  File "simple.py", line 69, in connect
  File "ssl.py", line 1, in wrap_socket
OSError: [Errno 12] ENOMEM

I understand that this is a memory issue (probably caused by fragmentation), but why does the traceback point to line 1 of ssl.py ? This line is the import of the TLS library:
import tls

The classes wrap_socket() function which causes the error starts at line 33 of ssl.py .

So I am wondering why the import of the TLS lib would cause memory error. It should have been included already when creating ssl.SSLContext(). If I understand this correctly, the whole point of using ssl.SSLContext() is that we don't have to recreate this for each new connection attempt. Therefore I don't expect memory error when connect() and wrap_socket() is called again.

Can anyone please shed some light on this?

[Carglglz]

I understand that this is a memory issue (probably caused by fragmentation), but why does the traceback point to line 1 of ssl.py ?

this is due to ssl.py byte code optimization level
see https://docs.micropython.org/en/latest/library/micropython.html#micropython.opt_level

Source-code line numbers: at levels 0, 1 and 2 source-code line number are stored along with the bytecode so that exceptions can report the line number they occurred at; at levels 3 and higher line numbers are not stored.

[hoihu]

I've had a similar issue using an API with https on the raspberry pi pico W.

When I issued the requests, it would always fail with the ENOMEM fault, using the urequests module.

This beside gc.mem_free reporting close to 100kB of free space.

Digging into the mbedtls lib, it seems that this lib needs 16kB of continous heap. Such a block can sometimes be hard to obtain by Micropython due to fragmentation.

The only dirty workaround for that is to do a something like that:

# allocate a large enought buffer for SSL connections, early in main.py
dummy = bytearray(20000)

...
...
# before issuing a request, free the buffer
del dummy
gc.collect()
r = urequests.get('https://api.com/example`)
r.close()
dummy = bytearray(20000)

basically it allocates a large buffer early at startup, frees it, issues the request, then re-claims it again.

I'm aware this is a very bad workaround!

What's missing in the ssl module is a method to assign a preallocated buffer or similar. A quick glance in the mbdedtls docs gave me the impression that it supports a preallocated memory area: https://mbed-tls.readthedocs.io/en/latest/kb/how-to/using-static-memory-instead-of-the-heap/#using-the-mbed-tls-buffer-allocator - not sure if that one could be used...

[ondrej1024]

I understand that this is a memory issue (probably caused by fragmentation), but why does the traceback point to line 1 of ssl.py ?

this is due to ssl.py byte code optimization level see https://docs.micropython.org/en/latest/library/micropython.html#micropython.opt_level

Source-code line numbers: at levels 0, 1 and 2 source-code line number are stored along with the bytecode so that exceptions can report the line number they occurred at; at levels 3 and higher line numbers are not stored.

Thanks, this is good to know. However, the trace should probably report the line number as "unknown" instead of 1 which is entirely misleading.

[ondrej1024]

What's missing in the ssl module is a method to assign a preallocated buffer or similar. A quick glance in the mbdedtls docs gave me the impression that it supports a preallocated memory area: https://mbed-tls.readthedocs.io/en/latest/kb/how-to/using-static-memory-instead-of-the-heap/#using-the-mbed-tls-buffer-allocator - not sure if that one could be used...

The SSL implementation of the latest MP release uses the SSLContext() function. As I understand it, this function was introduced to allocate the memory used for SSL handling only once during initialization and reuse it each time a connection needs to be reestablished. So I was hoping that this would fix the memory issues during reconnection. Unfortunately this isn't the case.

[ondrej1024]

The root cause of this issue was discovered. A fix has been implemented by @projectgus :
#16015