Fixes #36833 - Add SecureBoot support for arbitrary operating systems to "Grub2 UEFI" PXE loaders #877
Conversation
|
Can one of the admins verify this patch? |
|
I have opened a PR to document this feature in foreman-documentation: |
|
This PR also requires the changes from Foreman PR #9864 |
There was a problem hiding this comment.
I'd advertise this as an optional capability. In https://theforeman.org/2019/04/smart-proxy-capabilities-explained.html I've written about this. The benefit is two fold. On the one hand, it can disabled if bootloader_universe is not set. Foreman can detect this. It can also help with n-1 support where Foreman x.y runs with a Foreman Proxy x.(y-1).
One example where you can see it in Foreman is https://github.com/theforeman/foreman/blob/66f0c87b26c9b3a125bd216eb26c9a28b06effb3/app/models/concerns/orchestration/dhcp.rb#L84-L86 but Katello also uses it for the various content types.
|
I encountered an error on a standalone Smart Proxy with the check if the bootloader directory is configured and fixed it. |
|
Hi there, What's left to do for now:
@sbernhard @ekohl What do you think? Please share your thoughts. I'm looking forward to input and discussion. |
|
Implemented changes as discussed in the course of the Foreman Community Forum thread. Please let me know, if anything is missing or further changes are required. |
|
Added deletion of host specific bootloader files in case host leaves build mode. |
|
[test smart-proxy] |
|
ok to test |
… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds a new parameter 'bootloader_universe' to the TFTP configuration and a directory 'host_config' inside the TFTP root directory, that are both required by the aforementioned PRs.
|
Squashed and changed commit message to reflect current implementation. => Any objections against merging this? What do you think, @nofaralfasi and @stejskalleos? |
goarsna
changed the title
pr-processor
bot
added
Needs re-review
and removed
Waiting on contributor
Needs re-review
labels
There was a problem hiding this comment.
Two more notes from my side:
-
Since we are no longer sending the
buildparameter in the POST request when initializing the TFTP service, we follow the same procedure when the host enters build mode and when it exits build mode.
IMO, there's no need to delete the existing symlinks and create new ones when the action performed is to disable build mode for a host. -
We currently lack tests for the new workflow. It would be great to have at least some tests for the symlink generation.
|
Hey Nofar, once again thanks for your feedback! Regarding the two questions from your comment above:
Indeed the PR is lacking tests 🙈 I will think about that and add some.
I wasn't able to finish my research on that and I am off on PTO until Monday, October 10th. But here are my current thoughts: |
I am not in the technical details here, but: the difference between the two build modes (enabled/disabled) basically happens in the GRUB2 configuration file. Thus, symlinks to the correct OS related network boot files of the host is mandatory for installation (=build mode enabled) and booting from disk (=build mode disabled; loading local GRUB2 configuration from local disk). The only event which should trigger creation of new symlinks is when the OS of the host changes, ideally independent of any build mode switch, but at latest when switching build mode. And to make it easier, this can happen in both switch cases for simplicity. What we could think about is, if not already considered/implemented, to symlink also all the GRUB2 configuration files to avoid duplicates which might be confusing (e.g. |
Thank you for pointing that out. So, what you're suggesting is that new symlinks should be created even when the host is not in build mode, as this might still be necessary? For instance, when a user updates the OS directly on the host and then updates Foreman to reflect the new version, the symlinks would still need to be updated. Please correct me if I’m misunderstanding.
I have no objections to doing that, but perhaps we should keep it outside the scope of this PR. |
Correct. The described example by you is probably a corner case, but such an host wouldn't be able to reboot because build mode wasn't changed at all, thus no symlinks have been updated at all. This means the former OS vendor shim tries to boot the new OS kernel from disk (requires theforeman/foreman#10247). Question is when the TFTP API is triggered. That's something you guys might now better. Is this only when switching build mode? Or also when a host was updated at all (edit host, click "Submit")? I just wanted to point out that whenever TFTP API is triggered, (re)writing symlinks should not harm, right? I don't think we need to distinguish if build mode was enabled or if build mode was disabled - just always (re)write symlinks. At the end it's always the same target if there wasn't any OS change for the host.
Yes, we can and should handle this separately. |
When build mode is changing, and also when updating a host.
I don’t think it should cause any harm. That's exactly what I want to clarify in this discussion. And I want to ensure that we're not performing unnecessary operations simply because they aren't harmful, but rather that there’s a valid reason for them. |
Hm, IMHO I could live with only (re)writing symlinks when build mode changes because I think it's not a valid use case to change the OS directly on the host. But as said, I think we do not need to distinguish between transitions (build mode enabled→disabled or disabled → enabled). And of course if the PXE loader changes (which is part of host update, right), then we would also need to (re)write symlinks. |
Why do we need to rewrite symlinks when the PXE loader changes? The symlinks are created for all hosts, even those that don’t use the Secure Boot PXE loaders, so it seems that changes to the PXE loader shouldn't have an impact. However, if we choose to rewrite the symlinks regardless of whether the host is in build mode, then this concern may be less significant. |
|
I’d like to summarize our discussion so far: With this approach, we rewrite a host’s symlinks every time it is updated, regardless of whether the host is in build mode. Our concern is that this adds unnecessary complexity to the workflow. However, there may be scenarios where this operation is needed, such as when changing the OS of a host without rebuilding it. @jloeser @goarsna @stejskalleos I’d appreciate your thoughts on this. |
|
Thanks, @jloeser and @nofaralfasi for all your input. I read through all your posts and I agree to what Jan wrote:
Yes, as the bootloader files depend on the operating system the symlinks should get updated when the operating system is changed and not depending on the build mode flag. Regarding your example, Nofar:
Jan said that this is a corner case, which may be true for most OSes. But if I remember correctly @lzap (please correct me in case I am wrong) mentioned once in a call that for Red Hat Enterprise Linux the shim has to match the EL minor version. So for RHEL it is necessary to redeploy the boot files on an OS change.
=> Without updating the symlinks, the host won't be able to boot anymore as it still consumes the RHEL 8.10 boot files. Also IMO updating the symlinks every time the TFTP configuration is redeployed should not harm. But I also agree with what Nofar said: We should not perform unnecessary operations just because they do not harm. So the best would be to only update the symlinks in the cases we require them to get updated and this is when the OS of a host is changed. And a check if the OS has been changed is already implemented in the queue_tftp_update method in Foremans tftp.rb#R169. I didn't take a deep look at the code for now but my first thought was that we could reuse the check in the proxy.set method called in tftp.rb#R90-R96 and extend it to also evaluate to Hui... This was a lot of text now... Sorry for that 😅 😄 Edit: Ok, I should add a small summary as my post was really pretty long:
What do you think, @jloeser and @nofaralfasi? And finally:
I like your suggestion and would be happy to see / do that in a follow-up PR. :) |
|
After posting I kept on thinking about further possible corner cases and a second scenario came to my mind where we (or better said the users) have to update the symlinks (If I recall correctly we once already talked / wrote about this case for short, but I want to bring it up here again to be aware of it): I am thinking about the case that certificates get revoked and a host suddenly has to use boot files from a OS- and version-specific directory because we want to boot newer hosts from the OS-specific default directory. Lets have a look at following fictional example:
=> The AlmaLinux 8.10 host will still boot using the symlinks that point to the But this case is a bit tricky as we have no change to the host in Foreman to determine that the symlinks have to be updated. The best would be to be able to manually trigger a redeployment of the TFTP configuration to update the symlinks (or is there already a possibility to manually trigger this?!). But IMO if we want to provide an somehow automated way to do this, it should definitely be a follow-up task. For now, users have to manually adjust the symlinks in this case. |
|
@goarsna I completely agree with everything you’ve outlined here. Great catch on identifying these potential scenarios. |
|
@nofaralfasi Thanks! :D Then I will try to find time to implement the "Update the symlinks only on OS changes"-change ASAP :) Nevertheless input, thoughts, ack by @jloeser / @stejskalleos is still appreciated :) |
After discussing with the team, everyone agreed there's no need to add extra logic for updating the symlinks only when the OS changes. We can proceed with keeping things as they are, with no further changes needed. |
|
Added tests for the creation of the symlinks as discussed in DM with @nofaralfasi. |
|
Hey Nofar, thanks for your review of the tests. The reason I chose not to test the individual methods is simply that I thought about "how are the methods used" while writing the tests. I reworked the tests and test the individual methods now. |
… to "Grub2 UEFI" PXE loaders
This feature consists of four patches, one each for foreman,
smart-proxy, foreman-installer, and puppet-foreman_proxy.
This patch adds support for individual Network Bootstrap Programs (NBP)
in order to enable network based installations of SecureBoot enabled
hosts for arbitrary operating systems.
SecureBoot expects to follow a chain of trust from the initial boot of
the host to the loading of Linux kernel modules. The very first shim
that is loaded determines which distribution is allowed to be booted or
kexec'ed until next reboot.
Currently the "Grub2 UEFI SecureBoot" PXE loaders use NBPs provided by
the vendor of the Foreman/Smart Proxy host system. All hosts receive and
execute the same binary. On SecureBoot enabled hosts, this limits
installations to operating systems by the vendor of the Foreman/
Smart Proxy host system.
Providing shim and GRUB2 by the vendor of the operating system to be
installed allows Foreman to install any operating system on SecureBoot
enabled hosts over network.
To achieve this, the host's DHCP filename option is set to a shim/GRUB2
binary in a host specific directory based on their MAC address.
Corresponding shim and GRUB2 binaries are copied into that directory
along with the generated GRUB2 configuration files.
When provisioning a host, the Smart Proxy checks in a dedicated
directory inside the TFTP root - the so called "bootloader universe" -
if NBPs are present matching the operating system, operating system
version, and architecture of the host to be installed. If this is the
case, these NBPs are copied from the bootloader universe directory to
the host specific directory. If not, as a fallback the default NBPs
provided by the vendor of the Foreman/Smart Proxy host system are
copied from the `:tftproot:/grub2` directory to the host specific
directory.
Up to now, shim and GRUB2 binaries have to be retrieved and set up in
the bootloader universe directory manually according to the
documentation. An automatic way to provide OS dependent NBPs will be
added in future.
In case there are no NBPs present in the bootloader universe matching
the operating system, operating system version, and architecture of
the host to be installed, the behaviour of the "Grub2 UEFI" PXE
loaders does not change to the behavior prior to this feature.
Implementation notes:
---------------------
* To be future proof (e.g. to be able to provide NBPs in the bootloader
universe for other PXE loaders without running into any filename
conflicts) and for better structure, the PXE kind is prepended as a
first directory level inside the bootloader universe.
* The operating system version inside the bootloader universe consists
of the major and minor version (if applicable) of the operating system
separated by a dot (`.`). If no NBPs are configured for a specific
operating system version the fallback directory `default` is used.
* To simplify things on Foreman side in future, symlinks are used for
the shim (boot-sb.efi) and GRUB2 (boot.efi) binaries.
* Inside the TFTP root directory a new directory `host-config` is
created for storing all the host specific directories.
* Inside the TFTP root directory a new directory `bootloader-universe`
is created for storing all the OS specific boot files.
* For storage efficiency the shim and GRUB2 binaries from the
bootloader universe or the `:tftproot:/grub2` directory are
symlinked to the host specific directory.
Full example:
-------------
[root@vm ~]# hammer host info --id 241 | grep -E "(MAC address|Operating System)"
MAC address: 00:50:56:b4:75:5e
Operating System: AlmaLinux 8.9
[root@vm ~]# tree /var/lib/tftpboot/bootloader-universe/
/var/lib/tftpboot/bootloader-universe/
└── pxegrub2
└── almalinux
├── 8.9
│ └── x86_64
│ ├── boot.efi -> grubx64.efi
│ ├── boot-sb.efi -> shimx64.efi
│ ├── grubx64.efi
│ └── shimx64.efi
└── default
└── x86_64
├── boot.efi -> grubx64.efi
├── boot-sb.efi -> shimx64.efi
├── grubx64.efi
└── shimx64.efi
[root@vm ~]# hammer host update --id 241 --build true
[root@vm ~]# tree /var/lib/tftpboot/host-config
/var/lib/tftpboot/host-config
└── 00-50-56-a3-41-a8
└── grub2
├── boot.efi -> ../../../bootloader-universe/grubx64.efi
├── boot-sb.efi -> ../../../bootloader-universe/shimx64.efi
├── grub.cfg
├── grub.cfg-00:50:56:a3:41:a8
├── grub.cfg-01-00-50-56-a3-41-a8
├── grubx64.efi -> ../../../bootloader-universe/grubx64.efi
├── os_info
└── shimx64.efi -> ../../../bootloader-universe/shimx64.efi
[root@vm ~]# grep -B2 00-50-56-b4-75-5e /var/lib/dhcpd/dhcpd.leases
hardware ethernet 00:50:56:b4:75:5e;
fixed-address 192.168.145.84;
supersede server.filename = "host-config/00-50-56-b4-75-5e/grub2/boot-sb.efi";
[root@vm ~]# pesign -S -i /var/lib/tftpboot/host-config/00-50-56-b4-75-5e/grub2/boot-sb.efi | grep "Microsoft Windows UEFI Driver Publisher"
The signer's common name is Microsoft Windows UEFI Driver Publisher
This feature consists of two patches, one for foreman and one for smart-proxy.
This patch introduces a new loader of kind
:PXEGrub2TargetOSwhich allows to provide host-specific Network Bootstrap Programs (NPB) in order to enable network based installations for SecureBoot-enabled hosts.SecureBoot expects to follow a chain of trust from the start of the host to the loading of Linux kernel modules. The very first shim that is loaded basically determines which distribution is allowed to be booted or kexec'ed until next reboot.
The existing "Grub2 UEFI SecureBoot" is not sufficiant as it limits the possible installations to the vendor of the Foreman (Smart Proxy) host system.
Providing shim and GRUB2 by the vendor of the to-be-installed operating system allows Foreman to install any operating system on SecureBoot-enabled hosts over network.
To achieve this, the host's DHCP filename option is set to a shim path in a directory that is host-specific (contains MAC address). Corresponding shim and GRUB2 binaries are copied into that directory along with the generated GRUB2 configuration files as we know from "Grub2 UEFI".
The required binaries must be provided once in the so called "bootloader universe". This directory can be configured via the settings file
/etc/foreman-proxy/settings.d/tftp.ymland defaults to/usr/local/share/bootloader-universe/<os>/. These binaries can be manually retrieved from the installation media and is not part of this patchset.Full example: