Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add pre-upgrade jobs to run helm-mapkubeapis (#1439) (2.5) #1454

Merged
merged 4 commits into from
Aug 4, 2023

Conversation

cbuto
Copy link
Contributor

@cbuto cbuto commented Aug 1, 2023

What problem does this PR solve?:
Add pre-upgrade jobs to kubecost, logging-operator, and gatekeeper to run helm-mapkubeapis tool that takes care of editing the helmrelease secret to remove the podsecuritypolicy manifest which is unsupported starting in k8s 1.25.

This affects clusters that were upgraded from 2.4, in which the PSP api was still supported and these 3 charts had deployed PSPs. Since kommander applications are upgraded prior to k8s, we saw this:

  • upgrade to kommander 2.5 --> these apps upgraded successfully, with PSP resources still deployed
  • upgrade to k8s 1.25 --> PSP api was removed, so all existing PSPs are deleted
  • upgrade to kommander 2.6 --> upgrade errors in these 3 apps, explanation here: https://helm.sh/docs/topics/kubernetes_apis/#helm-users

Helm fails in this scenario because it attempts to create a diff patch between the current deployed release (which contains the Kubernetes APIs that are removed in this Kubernetes version) against the chart you are passing with the updated/supported API versions. The underlying reason for failure is that when Kubernetes removes an API version, the Kubernetes Go client library can no longer parse the deprecated objects and Helm therefore fails when calling the library. Helm unfortunately is unable to recover from this situation and is no longer able to manage such a release.

The following section https://helm.sh/docs/topics/kubernetes_apis/#updating-api-versions-of-a-release-manifest tells us how to recover from this error -- including using this tool https://github.com/helm/helm-mapkubeapis to help us do it easily via a single command. We have included this plugin into the kubetools image: https://github.com/mesosphere/kommander/pull/3657 and are using this image in these newly added pre-upgrade jobs.

Which issue(s) does this PR fix?:

https://d2iq.atlassian.net/browse/D2IQ-98509

Special notes for your reviewer:

Does this PR introduce a user-facing change?:


Checklist

  • If the PR adds a version bump, ensure there is no breaking change in Licensing model (or NA).
  • If a chart is changed or app configuration is significantly changed, the chart version is correctly incremented (so that apps are not automatically upgraded from a previous version of DKP).

@cbuto cbuto self-assigned this Aug 1, 2023
@cbuto cbuto added ok-to-test Signals mergebot that CI checks are ready to be kicked off backport-pr This is a backport PR labels Aug 1, 2023
@github-actions github-actions bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 1, 2023
@d2iq-mergebot
Copy link
Contributor

This repo has @d2iq-mergebot integration. You can perform the following commands by submitting a comment. Submit a comment with content "@d2iq-mergebot help" to view more detailed help text and examples. Be sure the have a look at the mergebot documentation, too.For help using mergebot, please refer to the README file here: https://github.com/mesosphere/mergebot/blob/main/README.md
Enabled Mergebot commands:
@d2iq-mergebot test all
@d2iq-mergebot test
@d2iq-mergebot override-status
@d2iq-mergebot help
@d2iq-mergebot backport

* feat(kubecost): Add pre-upgrade to run helm-mapkubeapi

* feat(kubecost): Use kommander chart version for kubetools image tag

* feat(kubecost): Use force true on pre-upgrade job

* feat(logging-operator): Add pre-upgrade job to run helm-mapkubeapis

* feat(gatekeeper): Add pre-upgrade job to run helm-mapkubeapis

* fix(gatekeeper): update release name

* fix(gatekeeper): add dependsOn in HR kustomization

* feat: Update release script to update the kubetools image tag

* chore: clean up todos

* refactor: typo

* fix: check if HR exists in pre-upgrade scripts

---------

Co-authored-by: Grace Do <xgrace@gmail.com>
@cbuto
Copy link
Contributor Author

cbuto commented Aug 2, 2023

I've tested:

  • upgrade from 2.4 to 2.5.2 (fixed by disabling the PSPs explicitly on upgrade) ✅
  • upgrade from 2.4 to 2.5.1 to 2.5.2 ✅
  • installs for 2.5.2 ✅

The CI checks here fail because of the version agnostic CLI

@cbuto cbuto marked this pull request as ready for review August 2, 2023 20:19
@cbuto cbuto requested a review from a team August 2, 2023 20:19
@cbuto cbuto merged commit 4acebf4 into release-2.5 Aug 4, 2023
7 checks passed
@cbuto cbuto deleted the cbuto/psp-fix branch August 4, 2023 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-pr This is a backport PR ok-to-test Signals mergebot that CI checks are ready to be kicked off ready-for-review size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants