New approach to squashfs images

[mattgodbolt]

Intention is to replace /opt/compiler-explorer entirely with something like this! at least configurably; should allow for both regular ce_install install .. backed by real files and supermagic squashfs things, pretty much transparently.

Very much WIP! But I got this PoC working well enough to see promise. Ideally I'll run with a whole base image and see what we get.

This does mean we rely entirely on the yaml files to install everything "from scratch" when we rebuild. We may not want this but for dicsussion!

[partouf]

Because my /opt/compiler-explorer is filled with other stuff like CE git repo's, I did sudo ln -sfT ~/ce /opt/cenew instead, and use ce_install --dest /home/partouf/ce buildroot gcc instead of leaving it at the default. Seems to work fine.

Question;

• There are 2 sqfs files per installation, what's the small one, is that the actual metainfo?
• If I try to 'buildroot' something that doesn't exist, it squashes ... something... twice, and puts 1 extra file in the /opt/cefs-images folder, I don't think it needs to do that, right?
• buildroot does seem to work nicely if you want to add all the compilers of a certain maker, it stores it in just 1 sqfs file, very cool
• Would we prepare a root sqfs and then hardcode the root hash into the AMI for instances?
• What about old compilers that we can't reinstall anymore, can we copy squash them over manually?
• Ubuntu Files window seems to kick me out of /opt/cenew and /home/partouf/ce every X minutes, (and throws me to the parent directory) is there some timer in the autofs thing that causes this, and how would that manifest in reality? I hope this would not mean that during compiler execution all of the things not paged in suddenly disappear for a second and half the files seem to be gone......

[mattgodbolt]

Because my /opt/compiler-explorer is filled with other stuff like CE git repo's, I did sudo ln -sfT ~/ce /opt/cenew instead, and use ce_install --dest /home/partouf/ce buildroot gcc instead of leaving it at the default. Seems to work fine.

Cool! In this case you could have "just" used /home/partouf/ce as the dest and ignored the /opt/. But also the code lets you specify the /opt and it follows the symlinks to the right place. But either way, glad it worked out!

Question;

• There are 2 sqfs files per installation, what's the small one, is that the actual metainfo?

Each installation will be at least 2 sqfs files:

• One sqfs layer that containes only the updated/freshly installed files (like a docker layer)
• As many (pre-existing) sqfs layers to link to the unchanged content
• One sqfs that contains the symlinks to everything.

To make it concrete, something like:

• assuming currently you have one layer with SHA basedata and root SHA root0
  • root0 contains symlinks for all directories that point into /cefs/basedata/ eg /cefs/basedata/gcc
• you install "clang" with buildroot (terrible name, it's really "install")
• a new layer newdata is created and clang is installed into that
• a new root root1 will be created with symlinks from gcc->/cefs/basedata/gcc and clang->/cefs/newdata/clang

The old root root0 is now unused and could be GC'd (but it's only small).

The plan is to never have more than a few layers around, with a weekly consolidation process that "flattens" down images into a single base layer again. Or something like that.

• If I try to 'buildroot' something that doesn't exist, it squashes ... something... twice, and puts 1 extra file in the /opt/cefs-images folder, I don't think it needs to do that, right?

Correct: the buildroot (currently) doesn't spot that nothing changed, and so it creates an empty data layer, and then makes a new root image (which only differs in the metadata). The new root image will point at only the previous sqfs files, not the empty one.

• buildroot does seem to work nicely if you want to add all the compilers of a certain maker, it stores it in just 1 sqfs file, very cool

Right! The hope is to literally put everything in one sqfs. And then actually squash the daliy builds in an overlay so there's one unified way of seeing all the compilers etc. No more "hack the squashfs" stuff. And if we rebuild something we can buildroot --force to put it in a new layer.

• Would we prepare a root sqfs and then hardcode the root hash into the AMI for instances?

We have some choices:

• Leave the /opt/compiler-explorer on EFS and make that a symlink to /cefs/. We can then atomically "just" change the symlink and instantly everyone gets the new image. Makes for simple deploys!
• Make it a setting like the "current build" on S3. AMIs grab the "root hash" from S3 at startup and then symlink their own /opt/compiler-explorer there. Updating the builds then is pushing a new "root image" doc at S3 (much like ce builds set_current) and then refresh everything.
• Baking it into an AMI. which would then mean daily AMI builds or after each installation.

• What about old compilers that we can't reinstall anymore, can we copy squash them over manually?

Yeah this is a great point. I was going to make a cefs copy $list-of-files type thing to let us manually copy things into the base image. But really I am interested to see how well we do installing from scratch too :) We will need this though, you're right. And the "consolidate" process will probably have to unpack the whole images and reassemble, rather than reinstalling from scratch (which was my original thought). For discussion!

• Ubuntu Files window seems to kick me out of /opt/cenew and /home/partouf/ce every X minutes, (and throws me to the parent directory) is there some timer in the autofs thing that causes this, and how would that manifest in reality? I hope this would not mean that during compiler execution all of the things not paged in suddenly disappear for a second and half the files seem to be gone......

Yes! autofs is (deliberately) configured with a timeout here so it unmounts stuff pretty aggressively. We can of course change that, but the value here is one that's battle-tested in $dayjob so it should be ok for most practical purposes (if what you're seeing is indeed timeouts).

[partouf]

Re: baking it into an AMI. Daily builds would be fine, but I would not be looking forward to using terraforming as a deploy mechanism. So maybe just fetching the hash from S3 is the best option here.

[mattgodbolt]

I lied about tuning: The default is 10 minutes. we're using the default here for idly unmounting squashfs after 10m.

[partouf]

I lied about tuning: The default is 10 minutes. we're using the default here for idly unmounting squashfs after 10m.

I see. Can we maybe turn that off for production if we find out that causes issues?

[apmorton]

I lied about tuning: The default is 10 minutes. we're using the default here for idly unmounting squashfs after 10m.

I see. Can we maybe turn that off for production if we find out that causes issues?

It shouldn't cause issues - any open file handle inside the mount is enough to keep it mounted (and also bump the timeout forward). We use this exact autofs setup at $dayjob.

The auto unmount is what allows "magic" atomic swaps with no cleanup.

"Just" swap out the symlink that points to the new autofs mount point and everybody picks up the new stuff during their next compile, and then the old mounts die after a timeout.

[partouf]

I lied about tuning: The default is 10 minutes. we're using the default here for idly unmounting squashfs after 10m.

I see. Can we maybe turn that off for production if we find out that causes issues?

It shouldn't cause issues - any open file handle inside the mount is enough to keep it mounted (and also bump the timeout forward). We use this exact autofs setup at $dayjob.

The auto unmount is what allows "magic" atomic swaps with no cleanup.

"Just" swap out the symlink that points to the new autofs mount point and everybody picks up the new stuff during their next compile, and then the old mounts die after a timeout.

Ok I see. But if you're super unlucky and it's a quiet day and you hit an instance that hasn't had activity for 10 minutes, you have a nanominuscule chance that you connect during an unmount/mount cycle. Probably fine.

[apmorton]

I lied about tuning: The default is 10 minutes. we're using the default here for idly unmounting squashfs after 10m.

I see. Can we maybe turn that off for production if we find out that causes issues?

It shouldn't cause issues - any open file handle inside the mount is enough to keep it mounted (and also bump the timeout forward). We use this exact autofs setup at $dayjob.
The auto unmount is what allows "magic" atomic swaps with no cleanup.
"Just" swap out the symlink that points to the new autofs mount point and everybody picks up the new stuff during their next compile, and then the old mounts die after a timeout.

Ok I see. But if you're super unlucky and it's a quiet day and you hit an instance that hasn't had activity for 10 minutes, you have a nanominuscule chance that you connect during an unmount/mount cycle. Probably fine.

We can avoid this my moving the health check file inside the sqfs root - similar to how we currently have the health check on nfs.

Every ELB health check would resolve the symlink and ensure the most recent root is mounted.

[mattgodbolt]

We can avoid this my moving the health check file inside the sqfs root

but what's true for the root is also true for all the data images too? And we've never seen this at dayjob: I wonder if there's something specifically odd going on with shells left cd'd into a directory that doesn't trigger things.

[partouf]

We can avoid this my moving the health check file inside the sqfs root

but what's true for the root is also true for all the data images too? And we've never seen this at dayjob: I wonder if there's something specifically odd going on with shells left cd'd into a directory that doesn't trigger things.

that could be, my observation was about this view:

which will change automagically to my home in 10ish minutes.

[partouf]

5-6ish minutes in this case

[mattgodbolt]

Thanks for the info partouf. I am 99.5% sure that's an artifact of nautilus not holding the directory file descriptor open (on purpose!) and then detecting the unmount that autofs does, and closing the window (or similar). If it affects your workflow we can bump the timout to a day or so or whatever, but without a timeout we'll slowly accumulate mounts (which might be ok for local dev?)

Separately to this: I've been trying to tidy up the nomenclature. For more PoC stuff I'll need to do:

• importing of existing directories (e.g the one big import of /opt/compiler-explorer as it stands now)
• consolidating of images
• gc of unused images

Once I've tried that out I'll write it up more and we can discuss. Maybe on Friday?!