fix(system-update): reboot reliably after a kernel update - #12
Open
deepak7340 wants to merge 1 commit into
Open
Conversation
Two bugs kept nodes on an old kernel after system-update installed a new one. 1. getInstalledKernel piped `find /boot/vmlinuz-* | sort -V | tail -1` to pick the newest kernel. GNU version sort ranks the RHEL point release `4.18.0-553.el8_10` above every z-stream update (`4.18.0-553.157.1.el8_10`, ...) because after `553.` it compares a letter as greater than a digit. A node booted on the point release therefore compared equal to "newest" and never rebooted. Replace the shell pipeline with a filepath.Glob plus an rpmvercmp-style comparison (compareKernelVersions) that treats a numeric segment as newer than an alphabetic one, matching the package managers. 2. The reboot was fire-and-forget. linuxaid-cli issued `reboot --force` and returned immediately; obmondo-system-update.service is Type=oneshot, so systemd marked the unit finished and tore down its cgroup, killing the pending reboot before PID 1 acted (seen in the journal as the reboot log line and "Deactivated successfully" in the same second, box still up). Now the reboot request's error/exit code is checked and the call blocks until the machine goes down; if the grace period elapses it reports the failure instead of returning silently. Also log every branch of CheckKernelAndRebootIfNeeded: a silent no-op was what made the first bug invisible in the journal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs kept nodes on an old kernel after system-update installed a new one.
getInstalledKernel piped
find /boot/vmlinuz-* | sort -V | tail -1to pick the newest kernel. GNU version sort ranks the RHEL point release4.18.0-553.el8_10above every z-stream update (4.18.0-553.157.1.el8_10, ...) because after553.it compares a letter as greater than a digit. A node booted on the point release therefore compared equal to "newest" and never rebooted. Replace the shell pipeline with a filepath.Glob plus an rpmvercmp-style comparison (compareKernelVersions) that treats a numeric segment as newer than an alphabetic one, matching the package managers.The reboot was fire-and-forget. linuxaid-cli issued
reboot --forceand returned immediately; obmondo-system-update.service is Type=oneshot, so systemd marked the unit finished and tore down its cgroup, killing the pending reboot before PID 1 acted (seen in the journal as the reboot log line and "Deactivated successfully" in the same second, box still up). Now the reboot request's error/exit code is checked and the call blocks until the machine goes down; if the grace period elapses it reports the failure instead of returning silently.Also log every branch of CheckKernelAndRebootIfNeeded: a silent no-op was what made the first bug invisible in the journal.