Executing natmod .mpy from flash

[haydenroche5]

Hi,

I'm one of the main developers of note-python, a Python library for communicating with a Blues Notecard. note-python supports regular old CPython, MicroPython, and CircuitPython. We have an analogous library written in C, note-c.

At this moment, note-python and note-c are completely separate projects. When we implement a feature in note-c, we eventually come around and implement it in note-python. For awhile, we've wanted to be able to consume note-c in note-python such that when we add a feature to note-c, we get it (mostly) for free in note-python. This isn't terribly difficult in the CPython case via CFFI.

I've been hacking on a PoC of this approach for MicroPython. One major advantage of note-python as it exists now is that it doesn't require the user to rebuild and flash new MicroPython firmware onto their target. The user can simply copy note-python over to the target's filesystem and begin using it. In order to retain as much of this convenience as possible, I went down the "natmod" path. This approach will require us to distribute architecture-specific .mpy files for note-python, but it won't require a rebuild and reflash of MicroPython firmware. So far, I've managed to communicate with a Notecard over I2C using note-c in the resulting .mpy file, which is great. However, as I've begun to pull in more note-c functionality, the .mpy file has grown, and now when I attempt to import it, I'm running out of RAM for the code (i.e. MemoryError: memory allocation failed, allocating 24920 bytes).

So that got me thinking, is there a way to get this code into the flash storage of the target instead? There's the "extmod" approach, but if we go that route, we'll have to get our users to build and flash new MicroPython firmware to use note-python, which is a potential dealbreaker.

I've seen several GH issues and some MicroPython forums archive posts over the years discussing this topic, but, from what I can tell, there's no way to do what I'm trying to do, yet (i.e. execute a natmod .mpy from flash).

I would love any thoughts on how I might approach this problem.

Thanks!

[andrewleech]

Interesting application!
That error you're getting, trying to allocate nearly 25KB, that's quite a big chunk! Is your mpy that big, or is it allocating buffers or something during import?

In terms of loading directly from ram, the only existing methods I know involve freezing the code into the firmware, which requires compiling the codebase. I'm pretty sure that doesn't currently work with natmods either.

I suspect the feature you're pretty much looking for is #8381 which is intended to allow loading files of any type onto a "virtual" filesystem such that it can be flashed separately to a standard firmware download.
End users would be able to use the included tool to load your library and any other "core" libraries they need into a static filesystem fine and flash that, without needing a compiler or anything.

[haydenroche5]

Hi @andrewleech . Thanks for taking time to check out my question.

That error you're getting, trying to allocate nearly 25KB, that's quite a big chunk! Is your mpy that big, or is it allocating buffers or something during import?

That's on import, and I think I've figured out what's taking up so much space, after reading this comment from @jimmo:

The problem is that your dynamic module ends up having to bring in all the soft float support code, because it has no way to link to the support already provided in the main firmware. (The MicroPython dynamic linker can only link the symbols in dynruntime.h). Hardware float is fine because there is no support code required. (e.g. what you're seeing here is the linker error because it needs the soft implementation of float multiply)

For context, I'm targeting an ESP32 for my experiments. Indeed, I had to pull out a bunch of object files from libgcc.a and libc.a and add them to the build to get everything to link. This amounts to a major duplication of code, as all this stuff is already in the MicroPython firmware proper. I just have no way of linking against it in my .mpy.

In terms of loading directly from ram, the only existing methods I know involve freezing the code into the firmware, which requires compiling the codebase. I'm pretty sure that doesn't currently work with natmods either.

On that note, I went ahead and implemented my code as an "extmod", despite the drawbacks I mentioned in my original post. This went much more smoothly. I can communicate with a Notecard over I2C as with the natmod approach, and I can pull in all of note-c without running out of memory. I don't have to manually add in any .o files from libc.a or libgcc.a. I also don't have to modify the note-c source code at all; with the natmod approach, I had to move everything that would typically wind up in .data sections to .bss and initialize the relevant variables at runtime instead of compile-time (a limitation that's noted in the docs). So that's all great, but it's probably untenable for our use case to distribute a bunch of different MicroPython firmware binaries for many combinations of MicroPython version and target hardware. Asking our users to build MicroPython from source is also a non-starter, due to complexity of that process.

I suspect the feature you're pretty much looking for is #8381 which is intended to allow loading files of any type onto a "virtual" filesystem such that it can be flashed separately to a standard firmware download.

I saw that PR while I was researching. It seems like a potentially good middle ground between the two approaches I've tried so far. I do think that, even with this approach, I'd be duplicating a bunch of libc and libgcc code in my .mpy, which is unfortunate, but it might be unavoidable.

[andrewleech]

Your natmod mpy file, how big is it on disk? Even pulling in C float functions I wouldn't expect it to get that large.

I've done similar before with a stm natmod, pulling in (duplicating) heaps of stm hal code that already exists in micropython... but didn't see this kind of memory issue on import.

[haydenroche5]

The .mpy is 28477 bytes on disk. Here's the report from the end of running make ARCH=xtensawin:

arch:         EM_XTENSA
text size:    25686
rodata size:  2608
bss size:     244
GOT entries:  273
GEN note.mpy

And here's a list of all the libgcc and libc .o files I'm pulling in, plus their sizes:

12K     _addsubdf3.o
8.0K    _cmpdf2.o
8.0K    _divdf3.o
20K     _divdi3.o
4.0K    _fixdfsi.o
4.0K    _fixunsdfdi.o
4.0K    _floatdidf.o
4.0K    _floatsidf.o
8.0K    lib_a-memchr.o
8.0K    lib_a-memcmp.o
4.0K    lib_a-strchr.o
8.0K    lib_a-strcmp.o
8.0K    lib_a-strcpy.o
4.0K    lib_a-strlen.o
8.0K    lib_a-strncmp.o
8.0K    lib_a-strstr.o
20K     _moddi3.o
8.0K    _muldf3.o
16K     _udivdi3.o
20K     _umoddi3.o

[andrewleech]

Oh wow, ok those float libraries are much bigger than I'd expected.... yeah not much you can do to reduce that I guess other than switching to a better processor :-D

[jimmo]

@haydenroche5 Andrew is right that #8381 would be very useful unfortunately this does not support native code in .mpy files, only bytecode.

The problem is relocation. In theory it is possible to generate truly position-independent code (where references to both code and data are PI). i.e. it's more than just using -fPIC. I think -mno-pic-data-is-text-relative was the relevant bit, as well as storing the GOT address in a register. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=139630111

The problem was, I'm not sure this mode of code generation is possible for non-ARM targets like esp32/xtensa. (But maybe that's changed since we last looked at this)

See also:
https://mcuoneclipse.com/2021/06/05/position-independent-code-with-gcc-for-arm-cortex-m/
https://eli.thegreenplace.net/2011/11/03/position-independent-code-pic-in-shared-libraries

And also, like you say, this doesn't solve the duplication of e.g. the floating point routines...

[haydenroche5]

Thanks for this context @jimmo.

[vshymanskyy]

I think there's still an opportunity to overcome this limitation via VfsMap: #8381 (comment)