Have partition be sized to minimum to fit specified files

[dtgriscom]

As I expect a number of buildroot users are these days, we use genimage to generate the output file image. Problem: we must explicitly set a size value large enough to fit the built system. Too large, and we're making unwieldy images; too small and the build fails. Even worse, we'll have to tune this again and again as we add features to our product, increasing the size of the contained files.

Would it be possible for genimage to measure the size of the files that will go into the partition (specified using the rootpath property) and then make a partition that can contain those files with little waste?

[michaelolbrich]

Hmmm, we would could do that in the mountpoint handling. We would have to move that before calling setup(), but that should not be a problem. That's the easy part.
Calculation the actual size in a C program is not so nice. Do you have any suggestions for this?
Calling du and parsing the output is my best idea, but that's not very nice.

[dtgriscom]

I'm a "Google for a Solution" kind of guy, and here's what I found for "size of a folder in C":

https://stackoverflow.com/questions/21569578/how-to-get-total-size-of-a-folder-in-c-with-recursivity

A side issue is how precise you need to be; it's easy to think of ways of estimating the final size that end up with a too-small partition. Suggestion: a configuration setting for how much extra space gets added to the estimation.

[michaelolbrich]

The problem is getting the corner cases right. From a quick lock, I'm pretty sure that this will skip directories named ... for example.

About the size: some unknown additional space is always needed for meta data. In PTXdist for example I start with the size of the uncompressed tar and use 125% of that by default.

[dtgriscom]

Oy: if someone wants to name a directory ... (or ~, or or *?*?*?) then they get what they deserve...

You could solve this with documentation, saying that file trees containing unusually named files won't be able to use this feature.

[michaelolbrich]

No, if we do this, then we do it right. Documentation like that is just shifting blame, it doesn't solve any problems when they occur. And I certainly don't know the file names of all the files in the images I create. And if I need to create a fallback elsewhere in case genimage get's it wrong, then It's not worth the effort implementing this in genimage at all.

[dtgriscom]

Sounds right to me. Let me know if you want more feedback/suggestions/opinions.

[Zlika]

Hi!
It seems that this "percent" size feature is not available in the following use case:

image boot.vfat {
  vfat  {
    files = {
      "zImage",
      "dt.dtb"
    }
    file extlinux/extlinux.conf {
      image = "extlinux.conf"
    }
  }
  size = 100%
}

I have the following error:
mkdosfs: Attempting to create a too small or too large filesystem.

Is there a workaround to have this partition automatically sized to its minimum?
Thanks.

[michaelolbrich]

100% cannot work. None of the tools used here, allow specifying this. So we calculate the space used by the content.

However, there is always some overhead that depends on both the filesystem and the structure of the content (how many files, filenames, directory hierarchy, etc.). Genimage cannot calculate this overhead. So for the size calculation we only use the file content. So 100% will never work.

If you want to get as close as possible, then you need to experiment here. I use 120% most of the time. It's close enough for me and has not failed for me yet. If you go too low, then it may work once, but fail later when the content changes.

[Zlika]

Thanks for your answer, but on my use case I have the "mkdosfs: Attempting to create a too small or too large filesystem." whatever the percent value I use, even 200%.

[michaelolbrich]

That sounds like a bug. Please open an new issue and add the output of genimage with --loglevel=3

[Zlika]

I created #227.