avrdude is slow

[sprintersb]

avrdude is slow when is flashes files that have pages that consist only of 0xff.

Such pages can be skipped which speeds up flashing. This applies independent of whether an erase is performed.

This happens when a program has several parts located far away from each other. For example, a program that hosts version information at the end of flash, or that hosts .rodata at the end of flash.

When the program (elf) has multiple program headers that have their LMA in flash, then it is enough the just flash these program headers. When ihex or similar is used, it should be easy enough to check for all 0xff's in a page.

[stefanrueger]

Which programmer and which parts does this issue talk about?

AVRDUDE knows some 160 programmers and some 330 parts. It is not always the case that writing flash pages with only 0xff is a NOP: for example, some bootloaders or programmers serving parts with U/PDI interfaces can, and do, deploy page erase before writing. To complicate things further, some bootloaders, eg, optiboot, ignore a chip erase command. When AVRDUDE cannot rely on CE, it has to send pages with 0xff out of necessity to ensure the flash is erased where the uploaded file wants it to be. Not doing so would risk leaving the, eg, previously flashed secret Coca Cola recipe exposed.

There is some optimisation for 0xff pages when AVRDUDE can be sure the respective programmer carries out a CE and the protocol/programmer allows this. It is likely that not every optimisation for every programmer is taken, and we encourage a PR when further optimisation is possible in specific cases.

[sprintersb]

Which programmer and which parts does this issue talk about?

It's about all programmers and all devices.

When a sequence of 0xff is written with programmer X, then it would also be written with programmer Y. No?

The only dependence on device is what the page size is, because only hole pages can be skipped, not partial pages.

Detection of pages with only 0xff's in them should be feasible, and detecting such pages will be must faster that writing them with no effect.

[sprintersb]

And erasing is carried out by an erase cycle, not by writing 0xff's, as far as I understand.

Writing 0xff's is void, no matter if an erase cycle is issued or not (without an erase cycle, you cannot set bits that were 0 previously, but that applies to all values, not just 0xff.

As I said, the becaviour is independent of whether an erase cycle is issued. If it's not issued, that writing 0xff's wont't set bits that are 0 bevore.

[stefanrueger]

Writing 0xff's is void, no matter if an erase cycle is issued or not

Fist of all, we are talking flash memory here not eeprom or other memories. And then, some programmers hide the physical nature of NOR-memory. Take, for example, -c arduino that serves an optiboot bootloader. This bootloader deploys an SPM page erase cycle before writing the page. In this case the flash memory does not look like NOR-flash to AVRDUDE. Sending an 0xff-page actually sets bits even if they were not set before. This is because optiboot has decided to use SPM page erase before it writes a page. Optiboot could have been programmed so that flash memory actually looks like NOR-flash, but it wasn't. It is [edit: can be] a security risk not to send 0xff-pages: Assume you have secret data in flash of a part with the optiboot bootloader. The user wants to overwrite the secrets. Optiboot ignores CE, so the only way to overwrite the secrets is to send 0xff-pages, not? And how would that work if AVRDUDE decided to drop all 0xff-pages?

There are parts that allow external page erase, eg, the XMEGAs or modern AVR parts (those with UPDI interface). Here the physical programmer can erase pages before writing a page. Again, depending on the protocol and execution, flash does not necessarily look like NOR memory to AVRDUDE as this issue wrongly assumes.

Summarising, not all flash memory looks like NOR-flash via all programmers. AVRDUDE knows a lot about individual programmers and has (quite a few) optimisations under its hood. The one suggested by this issue is not amongst them for the reasons outlined above.

If you find a specific programmer that can be optimised, kindly submit a PR

[sprintersb]

Ok, thank you very much for the explantions.

[mcuee]

@sprintersb

If you do not have a specific case, I will close this issue.

[mcuee]

Convert this into a discussion.

[sprintersb]

I have one final question though.

When the layout of the program is like A-FF-B where A is cose at the start of program memory, B is at the end, and FF is a stretch of FF's between, then you are saying that it's important to write the FF's.

But then the program is like A-FF with FF up to the end, then avrdude won't write the final FF's. So why is avrdude allowed to ignore the FF's in that case?

[stefanrueger]

A-FF with FF up to the end, then avrdude won't write the final FF's. So why is avrdude allowed to ignore the FF's in that case?

There is an option -A that does not remove the final FFs. A for Arduino and its, what some consider buggy, optiboot bootloader.

The removal of final FFs was introduced early on in AVRDUDE's 23 year old history, long before optiboot, XMEGAs and UPDI parts, at a time when, indeed, the AVR world was much simpler. It made sense then, and still makes sense now, to remove final FFs as normally chip erase happens before uploading (and actually works with most programmers). So, if someone downloaded the flash contents of a small (say blink) application then the download would regularly be trailed by FF-pages. Removing them is the single most useful optimisation that AVRDUDE implements. It is wrong in the case of -c arduino with optiboot bootloaders though it rarely leads to problems as the not uploaded trailing pages are normally not used by the application. This is still a security problem in the optiboot case as the previously uploaded secret Coca Cola recipe remains (partially) in flash.

If you wish to speed up the A-FF-B cases you might consider preparing A.hex and B.hex and then uploading with one command -e -U A.hex -U B.hex that utilises multiple -U steps and won't touch the middle bit.

If you are sure the -c programmer in question actually carries out a CE and the programmer does not (or cannot) use a page erase and does not rely on address auto-increments then uploading FF-pages is not needed. Should the AVRDUDE code for that -c programmer not do that optimisation already, feel free to submit a corresponding PR.

[stefanrueger]

I just realised that -c arduino actually automatically disables the trailing FF removal.

[sprintersb]

I came across this with code for one of the new AVR128* devices, where text and data LMA are located at the start of flash and rodata is located in the last 32 KiB block. The ELF file containt three PHDRs, one for each respective memory region. Even though there is a big "hole" between the PHDRs LMA, avrdude would flash it.

I thought avrdude could be speed up; but I was unaware that avrdude actually has so little control over the commands that are actually send to the device.

[stefanrueger]

I thought avrdude could be speed up

It may well be that AVRDUDE flashing could be sped up for some use cases. However, that would have to be done in a more considered way than originally suggested. We rely on community contributions for problems, use cases, code and testing. Considering that AVRDUDE has drivers for 162 programmers and more than 330 parts that needs to be on a much finer granularity than "it is slow: make it faster". So, if you have a patch for a particular programmer or programmer type, a good use case we would like to hear about that.

[sprintersb]

No I don't have a patch. Anyway, thanks for all your time and explanations. Now it's clearer to me why avrdude is behaving in the way it does.