Fixes #36834 - Add SecureBoot support for arbitrary operating systems to "Grub2 UEFI" PXE loaders #9864

goarsna · 2023-10-16T07:21:13Z

This feature consists of two patches, one for foreman and one for smart-proxy.

This patch introduces a new loader of kind :PXEGrub2TargetOS which allows to provide host-specific Network Bootstrap Programs (NPB) in order to enable network based installations for SecureBoot-enabled hosts.

SecureBoot expects to follow a chain of trust from the start of the host to the loading of Linux kernel modules. The very first shim that is loaded basically determines which distribution is allowed to be booted or kexec'ed until next reboot.

The existing "Grub2 UEFI SecureBoot" is not sufficient as it limits the possible installations to the vendor of the Foreman (Smart Proxy) host system.

Providing shim and GRUB2 by the vendor of the to-be-installed operating system allows Foreman to install any operating system on SecureBoot-enabled hosts over network.

To achieve this, the host's DHCP filename option is set to a shim path in a directory that is host-specific (contains MAC address). Corresponding shim and GRUB2 binaries are copied into that directory along with the generated GRUB2 configuration files as we know from "Grub2 UEFI".

The required binaries must be provided once in the so called "bootloader universe". This directory can be configured via the settings file /etc/foreman-proxy/settings.d/tftp.yml and defaults to /usr/local/share/bootloader-universe/<os>/. These binaries can be manually retrieved from the installation media and is not part of this patch set.

Full example:

[root@vm ~]# hammer host info --id 241 | grep -E "(MAC address|Operating System)"
MAC address: 00:50:56:b4:75:5e
Operating System: Ubuntu 22.04 LTS

[root@vm ~]# hammer host update --id 241 --build true

[root@vm ~]# grep -B2 00-50-56-b4-75-5e /var/lib/dhcpd/dhcpd.leases
hardware ethernet 00:50:56:b4:75:5e;
fixed-address 192.168.145.84;
supersede server.filename = "grub2/00-50-56-b4-75-5e/shimx64.efi";

[root@vm ~]# pesign -S -i /var/lib/tftpboot/grub2/00-50-56-b4-75-5e/grubx64.efi | grep Canonical The signer's common name is Canonical Ltd. Secure Boot Signing (2021 v1)

theforeman-bot · 2023-10-16T07:21:17Z

Can one of the admins verify this patch?

theforeman-bot · 2023-10-16T07:21:18Z

Can one of the admins verify this patch?

theforeman-bot · 2023-10-16T07:21:18Z

Can one of the admins verify this patch?

goarsna · 2023-10-17T12:53:27Z

I have opened a PR to document this feature in foreman-documentation:
theforeman/foreman-documentation#2145

goarsna · 2023-10-20T08:42:21Z

This PR also requires the changes from Foreman Smart Proxy PR #877

app/models/concerns/pxe_loader_support.rb

app/models/concerns/orchestration/tftp.rb

app/models/concerns/orchestration/dhcp.rb

app/models/concerns/orchestration/tftp.rb

goarsna · 2023-12-06T13:43:28Z

Hi there,
I thought a lot about how to simplify things in the Orchestration::TFTP module. I introduced a PXE_KIND_MAP in pxe_loader_support.rb to map template kinds to the respective template kind that should be used for rendering. By this the changes in tftp.rb become smaller and simpler.

The other thing I was investigating is why the queue_tftp_create method in tftp.rb loops over all template kinds the chosen operating system supports? As I couldn't find a good reason for it after all investigations I did, I changed the implementation. Now only the template for the chosen template kind gets rendered, what allowed to drop the ifs and elses and makes this method also pretty short.

Any thoughts or objections?

goarsna · 2024-01-15T13:59:22Z

Hey there,
unfortunately the PXE loader mapping wasn't working. I fixed it now.
Also templates weren´t generated in case the adjusted template kind is changed. This is necessary if we omit the loop, as only the selected template kind gets rendered.

What do you think, @ekohl?

And I just saw, that there are some failing tests. 😅 I'll do that ASAP.

goarsna · 2024-01-25T07:50:40Z

Hey @ekohl,
I had a lot of discussions with people here at ATIX. And we figured out that the best approach is to keep the loop in queue_tftp_create and instead add the new PXE loader to the default template_kinds in operatingsystem.rb. By this everything works as expected and the introduced changes are minimized.

But we definitely require some feedback, weather this PR is going in the right direction and regarding the Smart Proxy capability (see my question above).

Thanks!

app/models/concerns/orchestration/tftp.rb

goarsna · 2024-02-15T08:45:32Z

Hi there,
I have rethought the whole concept of how we wanted to add SecureBoot support for arbitrary operating systems due to the discussions in Foreman Community Forum and came up with an idea how to completely integrate what we want into the existing PXEGrub2 PXE loaders.

What's left to do for now:

I will have to take a look at the tests after my changes.
As this approach generates MAC specific directories directly inside /var/lib/tftpboot/ the ownership of this directory has to be changed to user foreman-proxy.
One open request by @lzap was to provide differentiated bootloader files in the bootloader_universe not only by OS vendor but also by OS version.

@sbernhard @ekohl What do you think? Please share your thoughts. I'm looking forward to input and discussion.

The assumption was that `exit 1` in GRUB2 triggers a boot from the next bootdevice by the firmware and that the `chainloader` command is not working at all when SecureBoot is enabled (`lockdown=y`). These assumptions seems to be wrong. It looks like that distribution vendors patch GRUB2 differently which results in different behavior affecting these assumptions. Some support `chainloader` command, some do simply end up in the BIOS menu when using `exit 1`. As an alternative we can do a "chainload light" and only load the GRUB2 configuration file from local disk. This means that the PXE booted GRUB2 boots the actual kernel from local disk. For successful SecureBoot verification, the following changes are required: #9864 The proposed solution would also work when SecureBoot is disabled, however to avoid side effects I propose to only boot next device if SecureBoot is enabled (GRUB2 variable `lockdown=y` [2]).

goarsna · 2024-07-03T19:37:06Z

See discussion in smart-proxy PR for information on recent changes.

Affects "Grub2 UEFI" PXE loaders * PR in foreman: theforeman/foreman#9864 * PR in smart-proxy: theforeman/smart-proxy#877 * RFC: https://community.theforeman.org/t/add-secureboot-support-for-arbitrary-distributions/32601/1

The assumption was that `exit 1` in GRUB2 triggers a boot from the next bootdevice by the firmware and that the `chainloader` command is not working at all when SecureBoot is enabled (`lockdown=y`). These assumptions seems to be wrong. It looks like that distribution vendors patch GRUB2 differently which results in different behavior affecting these assumptions. Some support `chainloader` command, some do simply end up in the BIOS menu when using `exit 1`. As an alternative we can do a "chainload light" and only load the GRUB2 configuration file from local disk. This means that the PXE booted GRUB2 boots the actual kernel from local disk. For successful SecureBoot verification, the following changes are required: theforeman#9864 The proposed solution would also work when SecureBoot is disabled, however to avoid side effects I propose to only boot next device if SecureBoot is enabled (GRUB2 variable `lockdown=y` [2]).

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds a new parameter 'bootloader_universe' to the TFTP configuration and a directory 'host_config' inside the TFTP root directory, that are both required by the aforementioned PRs.

goarsna · 2024-08-05T12:04:02Z

Rebased to head of develop

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds a new parameter 'bootloader_universe' to the TFTP configuration and a directory 'host_config' inside the TFTP root directory, that are both required by the aforementioned PRs.

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds the 'bootloader-universe' and 'host-config' directories inside the TFTP root, that are both required by the aforementioned PRs.

goarsna · 2024-08-06T06:04:41Z

I renamed the host-config directory as decided in the course of the above mentioned discussion. Besides that I also cleaned up some changes that have already been obsoleted by prior discussions / comments in this PR.

… to "Grub2 UEFI" PXE loaders In the course of theforeman/foreman#9864 and theforeman/smart-proxy#877, SecureBoot support for arbitrary operating systems has been added to the "Grub2 UEFI" PXE loaders. This patch adds the 'bootloader-universe' and 'host-config' directories inside the TFTP root, that are both required by the aforementioned PRs.

goarsna · 2024-08-08T12:06:46Z

See this comment for the latest update.

goarsna · 2024-08-28T06:51:10Z

While reviewing the changes again, I discovered further obsoleted changes in the tftp_test.rb.

Affects "Grub2 UEFI" PXE loaders * PR in foreman: theforeman/foreman#9864 * PR in smart-proxy: theforeman/smart-proxy#877 * RFC: https://community.theforeman.org/t/add-secureboot-support-for-arbitrary-distributions/32601/1

… to "Grub2 UEFI" PXE loaders This feature consists of four patches, one each for foreman, smart-proxy, foreman-installer, and puppet-foreman_proxy. This patch adds support for individual Network Bootstrap Programs (NBP) in order to enable network based installations of SecureBoot enabled hosts for arbitrary operating systems. SecureBoot expects to follow a chain of trust from the initial boot of the host to the loading of Linux kernel modules. The very first shim that is loaded determines which distribution is allowed to be booted or kexec'ed until next reboot. Currently the "Grub2 UEFI SecureBoot" PXE loaders use NBPs provided by the vendor of the Foreman/Smart Proxy host system. All hosts receive and execute the same binary. On SecureBoot enabled hosts, this limits installations to operating systems by the vendor of the Foreman/ Smart Proxy host system. Providing shim and GRUB2 by the vendor of the operating system to be installed allows Foreman to install any operating system on SecureBoot enabled hosts over network. To achieve this, the host's DHCP filename option is set to a shim/GRUB2 binary in a host specific directory based on their MAC address. Corresponding shim and GRUB2 binaries are copied into that directory along with the generated GRUB2 configuration files. When provisioning a host, the Smart Proxy checks in a dedicated directory inside the TFTP root - the so called "bootloader universe" - if NBPs are present matching the operating system, operating system version, and architecture of the host to be installed. If this is the case, these NBPs are copied from the bootloader universe directory to the host specific directory. If not, as a fallback the default NBPs provided by the vendor of the Foreman/Smart Proxy host system are copied from the `:tftproot:/grub2` directory to the host specific directory. Up to now, shim and GRUB2 binaries have to be retrieved and set up in the bootloader universe directory manually according to the documentation. An automatic way to provide OS dependent NBPs will be added in future. In case there are no NBPs present in the bootloader universe matching the operating system, operating system version, and architecture of the host to be installed, the behaviour of the "Grub2 UEFI" PXE loaders does not change to the behavior prior to this feature. Implementation notes: --------------------- * To be future proof (e.g. to be able to provide NBPs in the bootloader universe for other PXE loaders without running into any filename conflicts) and for better structure, the PXE kind is prepended as a first directory level inside the bootloader universe. * The operating system version inside the bootloader universe consists of the major and minor version (if applicable) of the operating system separated by a dot (`.`). If no NBPs are configured for a specific operating system version the fallback directory `default` is used. * To simplify things on Foreman side in future, symlinks are used for the shim (boot-sb.efi) and GRUB2 (boot.efi) binaries. * Inside the TFTP root directory a new directory `host-config` is created for storing all the host specific directories. * Inside the TFTP root directory a new directory `bootloader-universe` is created for storing all the OS specific boot files. * For storage efficiency the shim and GRUB2 binaries from the bootloader universe or the `:tftproot:/grub2` directory are symlinked to the host specific directory. Full example: ------------- [root@vm ~]# hammer host info --id 241 | grep -E "(MAC address|Operating System)" MAC address: 00:50:56:b4:75:5e Operating System: AlmaLinux 8.9 [root@vm ~]# tree /var/lib/tftpboot/bootloader-universe/ /var/lib/tftpboot/bootloader-universe/ └── pxegrub2 └── almalinux ├── 8.9 │ └── x86_64 │ ├── boot.efi -> grubx64.efi │ ├── boot-sb.efi -> shimx64.efi │ ├── grubx64.efi │ └── shimx64.efi └── default └── x86_64 ├── boot.efi -> grubx64.efi ├── boot-sb.efi -> shimx64.efi ├── grubx64.efi └── shimx64.efi [root@vm ~]# hammer host update --id 241 --build true [root@vm ~]# tree /var/lib/tftpboot/host-config /var/lib/tftpboot/host-config └── 00-50-56-a3-41-a8 └── grub2 ├── boot.efi -> ../../../bootloader-universe/grubx64.efi ├── boot-sb.efi -> ../../../bootloader-universe/shimx64.efi ├── grub.cfg ├── grub.cfg-00:50:56:a3:41:a8 ├── grub.cfg-01-00-50-56-a3-41-a8 ├── grubx64.efi -> ../../../bootloader-universe/grubx64.efi ├── os_info └── shimx64.efi -> ../../../bootloader-universe/shimx64.efi [root@vm ~]# grep -B2 00-50-56-b4-75-5e /var/lib/dhcpd/dhcpd.leases hardware ethernet 00:50:56:b4:75:5e; fixed-address 192.168.145.84; supersede server.filename = "host-config/00-50-56-b4-75-5e/grub2/boot-sb.efi"; [root@vm ~]# pesign -S -i /var/lib/tftpboot/host-config/00-50-56-b4-75-5e/grub2/boot-sb.efi | grep "Microsoft Windows UEFI Driver Publisher" The signer's common name is Microsoft Windows UEFI Driver Publisher

goarsna · 2024-09-19T08:52:03Z

See this comment in the Smart Proxy PR regarding the recent changes.

The assumption was that `exit 1` in GRUB2 triggers a boot from the next bootdevice by the firmware and that the `chainloader` command is not working at all when SecureBoot is enabled (`lockdown=y`). These assumptions seems to be wrong. It looks like that distribution vendors patch GRUB2 differently which results in different behavior affecting these assumptions. Some support `chainloader` command, some do simply end up in the BIOS menu when using `exit 1`. As an alternative we can do a "chainload light" and only load the GRUB2 configuration file from local disk. This means that the PXE booted GRUB2 boots the actual kernel from local disk. For successful SecureBoot verification, the following changes are required: theforeman#9864 The proposed solution would also work when SecureBoot is disabled, however to avoid side effects I propose to only boot next device if SecureBoot is enabled (GRUB2 variable `lockdown=y` [2]). (cherry picked from commit aca4023)

nofaralfasi · 2024-10-28T13:18:00Z

/packit build

pr-processor bot added the Not yet reviewed label Oct 16, 2023

goarsna mentioned this pull request Oct 20, 2023

Fixes #36833 - Add SecureBoot support for arbitrary operating systems to "Grub2 UEFI" PXE loaders theforeman/smart-proxy#877

Open

ekohl reviewed Oct 20, 2023

View reviewed changes

app/models/concerns/pxe_loader_support.rb Outdated Show resolved Hide resolved

stejskalleos self-assigned this Oct 30, 2023

stejskalleos assigned ekohl and sbernhard and unassigned stejskalleos Nov 9, 2023

sbernhard reviewed Nov 17, 2023

View reviewed changes