Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add pre-upgrade jobs to run helm-mapkubeapis #1439

Merged
merged 11 commits into from
Aug 1, 2023
Merged

feat: Add pre-upgrade jobs to run helm-mapkubeapis #1439

merged 11 commits into from
Aug 1, 2023

Conversation

cbuto
Copy link
Contributor

@cbuto cbuto commented Jul 31, 2023 •
edited by gracedo
Loading

What problem does this PR solve?:
Add pre-upgrade jobs to kubecost, logging-operator, and gatekeeper to run helm-mapkubeapis tool that takes care of editing the helmrelease secret to remove the podsecuritypolicy manifest which is unsupported starting in k8s 1.25.

This affects clusters that were upgraded from 2.4, in which the PSP api was still supported and these 3 charts had deployed PSPs. Since kommander applications are upgraded prior to k8s, we saw this:

  • upgrade to kommander 2.5 --> these apps upgraded successfully, with PSP resources still deployed
  • upgrade to k8s 1.25 --> PSP api was removed, so all existing PSPs are deleted
  • upgrade to kommander 2.6 --> upgrade errors in these 3 apps, explanation here: https://helm.sh/docs/topics/kubernetes_apis/#helm-users

Helm fails in this scenario because it attempts to create a diff patch between the current deployed release (which contains the Kubernetes APIs that are removed in this Kubernetes version) against the chart you are passing with the updated/supported API versions. The underlying reason for failure is that when Kubernetes removes an API version, the Kubernetes Go client library can no longer parse the deprecated objects and Helm therefore fails when calling the library. Helm unfortunately is unable to recover from this situation and is no longer able to manage such a release.

The following section https://helm.sh/docs/topics/kubernetes_apis/#updating-api-versions-of-a-release-manifest tells us how to recover from this error -- including using this tool https://github.com/helm/helm-mapkubeapis to help us do it easily via a single command. We have included this plugin into the kubetools image: https://github.com/mesosphere/kommander/pull/3657 and are using this image in these newly added pre-upgrade jobs.

Which issue(s) does this PR fix?:

https://d2iq.atlassian.net/browse/D2IQ-98509

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Checklist

  • If the PR adds a version bump, ensure there is no breaking change in Licensing model (or NA).
  • If a chart is changed or app configuration is significantly changed, the chart version is correctly incremented (so that apps are not automatically upgraded from a previous version of DKP).

@github-actions github-actions bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 31, 2023
@d2iq-mergebot
Copy link
Contributor

This repo has @d2iq-mergebot integration. You can perform the following commands by submitting a comment. Submit a comment with content "@d2iq-mergebot help" to view more detailed help text and examples. Be sure the have a look at the mergebot documentation, too.For help using mergebot, please refer to the README file here: https://github.com/mesosphere/mergebot/blob/main/README.md
Enabled Mergebot commands:
@d2iq-mergebot test all
@d2iq-mergebot test
@d2iq-mergebot override-status
@d2iq-mergebot help
@d2iq-mergebot backport

@github-actions github-actions bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 1, 2023
@gracedo gracedo changed the title feat(kubecost): Add pre-upgrade to run helm-mapkubeapi feat: Add pre-upgrade jobs to run helm-mapkubeapi Aug 1, 2023
@gracedo gracedo added the ok-to-test Signals mergebot that CI checks are ready to be kicked off label Aug 1, 2023
@gracedo gracedo changed the title feat: Add pre-upgrade jobs to run helm-mapkubeapi feat: Add pre-upgrade jobs to run helm-mapkubeapis Aug 1, 2023
@gracedo gracedo marked this pull request as ready for review August 1, 2023 15:32
@cbuto cbuto removed the do-not-merge label Aug 1, 2023
@cbuto
Copy link
Contributor Author

cbuto commented Aug 1, 2023

upgrade tests failing due to kommander version constraint, i ran through it manually from 2.4 and it was successful.

@cbuto cbuto merged commit 48be13a into release-2.6 Aug 1, 2023
11 checks passed
@cbuto cbuto deleted the cbuto/kubecost-fix branch August 1, 2023 20:33
@github-actions
Copy link
Contributor

github-actions bot commented Aug 1, 2023

💔 All backports failed

Status Branch Result
❌ release-2.5 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 1439

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

cbuto added a commit that referenced this pull request Aug 1, 2023
@cbuto @gracedo
feat: Add pre-upgrade jobs to run helm-mapkubeapis (#1439)
8a8c9cd 
* feat(kubecost): Add pre-upgrade to run helm-mapkubeapi

* feat(kubecost): Use kommander chart version for kubetools image tag

* feat(kubecost): Use force true on pre-upgrade job

* feat(logging-operator): Add pre-upgrade job to run helm-mapkubeapis

* feat(gatekeeper): Add pre-upgrade job to run helm-mapkubeapis

* fix(gatekeeper): update release name

* fix(gatekeeper): add dependsOn in HR kustomization

* feat: Update release script to update the kubetools image tag

* chore: clean up todos

* refactor: typo

* fix: check if HR exists in pre-upgrade scripts

---------

Co-authored-by: Grace Do <xgrace@gmail.com>
cbuto added a commit that referenced this pull request Aug 1, 2023
@cbuto @gracedo
feat: Add pre-upgrade jobs to run helm-mapkubeapis (#1439)
77b6256 
* feat(kubecost): Add pre-upgrade to run helm-mapkubeapi

* feat(kubecost): Use kommander chart version for kubetools image tag

* feat(kubecost): Use force true on pre-upgrade job

* feat(logging-operator): Add pre-upgrade job to run helm-mapkubeapis

* feat(gatekeeper): Add pre-upgrade job to run helm-mapkubeapis

* fix(gatekeeper): update release name

* fix(gatekeeper): add dependsOn in HR kustomization

* feat: Update release script to update the kubetools image tag

* chore: clean up todos

* refactor: typo

* fix: check if HR exists in pre-upgrade scripts

---------

Co-authored-by: Grace Do <xgrace@gmail.com>
cbuto added a commit that referenced this pull request Aug 1, 2023
@cbuto @gracedo
feat: Add pre-upgrade jobs to run helm-mapkubeapis (#1439)
a7c98bd 
* feat(kubecost): Add pre-upgrade to run helm-mapkubeapi

* feat(kubecost): Use kommander chart version for kubetools image tag

* feat(kubecost): Use force true on pre-upgrade job

* feat(logging-operator): Add pre-upgrade job to run helm-mapkubeapis

* feat(gatekeeper): Add pre-upgrade job to run helm-mapkubeapis

* fix(gatekeeper): update release name

* fix(gatekeeper): add dependsOn in HR kustomization

* feat: Update release script to update the kubetools image tag

* chore: clean up todos

* refactor: typo

* fix: check if HR exists in pre-upgrade scripts

---------

Co-authored-by: Grace Do <xgrace@gmail.com>
cbuto added a commit that referenced this pull request Aug 2, 2023
@cbuto @gracedo
feat: Add pre-upgrade jobs to run helm-mapkubeapis (#1439)
6766589 
* feat(kubecost): Add pre-upgrade to run helm-mapkubeapi

* feat(kubecost): Use kommander chart version for kubetools image tag

* feat(kubecost): Use force true on pre-upgrade job

* feat(logging-operator): Add pre-upgrade job to run helm-mapkubeapis

* feat(gatekeeper): Add pre-upgrade job to run helm-mapkubeapis

* fix(gatekeeper): update release name

* fix(gatekeeper): add dependsOn in HR kustomization

* feat: Update release script to update the kubetools image tag

* chore: clean up todos

* refactor: typo

* fix: check if HR exists in pre-upgrade scripts

---------

Co-authored-by: Grace Do <xgrace@gmail.com>
cbuto added a commit that referenced this pull request Aug 4, 2023
@cbuto @gracedo
feat: Add pre-upgrade jobs to run helm-mapkubeapis (2.5) (#1454)
4acebf4 
* feat: Add pre-upgrade jobs to run helm-mapkubeapis (#1439)

* feat(kubecost): Add pre-upgrade to run helm-mapkubeapi

* feat(kubecost): Use kommander chart version for kubetools image tag

* feat(kubecost): Use force true on pre-upgrade job

* feat(logging-operator): Add pre-upgrade job to run helm-mapkubeapis

* feat(gatekeeper): Add pre-upgrade job to run helm-mapkubeapis

* fix(gatekeeper): update release name

* fix(gatekeeper): add dependsOn in HR kustomization

* feat: Update release script to update the kubetools image tag

* chore: clean up todos

* refactor: typo

* fix: check if HR exists in pre-upgrade scripts

---------

Co-authored-by: Grace Do <xgrace@gmail.com>

* chore: bump kubecost to 0.33.2

* refactor(gatekeeper): add comment

* chore: bump centralized-kubecost to 0.33.2

---------

Co-authored-by: Grace Do <xgrace@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikolajb mikolajb mikolajb approved these changes

@takirala takirala takirala approved these changes

@gracedo gracedo gracedo approved these changes

Assignees

@cbuto cbuto

Labels
auto-backport backport-to-release-2.5 ok-to-test Signals mergebot that CI checks are ready to be kicked off ready-for-review size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@cbuto @d2iq-mergebot @mikolajb @takirala @gracedo