Modernize Helm 4 HIP

hips/hip-0012.md

@@ -1,38 +1,225 @@
 ---
 hip: "0012"
 title: "Helm 4 Development Process"
-authors: [ "Matt Butcher <matt.butcher@microsoft.com>" ]
+authors: [ "Matt Butcher <matt.butcher@microsoft.com>", "George Jenkins <gvjenkins@gmail.com>", "Matt Farina" ]
 created: "2021-04-02"
 type: "process"
 status: "draft"
 ---
 
 ## Abstract
 
-The Helm 4 development lifecycle will be managed by a release engineer.
-It will be kicked off in October, 2021.
-The process will last not less than six months.
-There will be an active development phase, followed by an alpha period, a beta period, and a release candidate period.
-The release manager will determine when Helm 4 is ready to ship.
+
+Helm 4 is planned as the next major version release of Helm.
+With KubeCon/CloudNativeCon North America 2024 marking the five year anniversary since the successful Helm 3 release.
+The development of Helm for is strongly desired in order to continue to evolve Helm and its ecosystem.
+
+This HIP documents the requirements and process for building Helm 4.
+Describing the scope of Helm 4 changes.
+With particular concern to maintaining continuality in the Helm ecosystem.
+
+This HIP also aims for an open and productive process that enables Helm 4 to build on the success of Helm 3 and to continues to evolve Helm for the success of the Helm communitity.
+
+
+
+<!--
+
+Changes are to be managed by accompanying (lighter-weight) "Helm 4 proposal", or standard Helm HIP.
+
+To be approved by Helm maintainers.
+Once approved, enabling the related changes to be approved/merged by a release engineer.
+In order to ease the transtions for users, behavioural changes will need to be accompained by release notes and migration guides, etc.
+
+
+Preventing a protacted process that prevents the release of Helm 4 in a timely fashion (there will be a Helm 5 of course!)
+
+The release engineer will determine when Helm 4 is ready to ship. And once shipped, the defined deprecation phase for Helm 3 will begin.
+-->
 
 ## Motivation
 
-We do not currently have a documented process for kickstarting new major Helm versions.
-During the Helm 3 process, we drew criticism for not having such documentation.
-However, a "generalized" process that can span multiple major versions over multiple
-years is ambitious.
-So this proposal attempts to merely spell out a formal process for Helm 4.
+Helm 3 has been greatly sucessful.
+And Helm has built a significant ecosystem of the Helm tooling (CLI), charts, chart repositories and repos, the SDK, plugins, etc.
+The Helm project needs to continue to evolve.
+Kubernetes itself has continued to grow and change.
+And Helm needs to continue to evolve as well to continue to provide value to the Helm community.
+
+However, Helm has also accumulated debt.
+And Helm 3 today has the significant issue that many (good) changes are overly difficult to implement due to compatibility concerns: either breakage of behavioural functionality or API incompatibility.
+
+Helm 4 as a major release allows the project to make API and other breaking changes.
+That will enable the project to reduce technical debt and leverage some newer capabilities of Kubernetes.
+
+However, Helm 4 must maintain continuality in Helm ecosystem.
+It is mandatory that Helm 4 _must_ be largely functionally/feature compatible with Helm 3.
+Signifcant functional breakage will prevent users from easily migrating to Helm 4.
+Will at best delay Helm 4 uptake and incur cost to Helm's users.
+Or at worst splinter the Helm ecosystem.
+
+The project also seeks to timebox the Helm 4 release process.
+Given the limited amount of community resource to enact change(s), it is considered much more important to produce a Helm 4 release which can deliver significant benefits in a moderate timeframe.
+Than a protracted process which takes too long to deliver meaningful results.
+
+<!--
+In order to maintain coherency in the Helm ecosystem.
+Breakage of signifcant features/functionmality used by the Helm community, without sufficent migration options/instructions.
+
+As such, this projects desires scoping changes for Helm 4 for two main reasons:
+
+Additionally, it is desirable to focus on cleaning up and fixing signifcant issues held back by Helm 3 compatiblity guarentees
+
+However, while it would be great to deliver many new exiciting features with Helm 4.
+
+
+Significantly important requirements for Helm 4 is to refactor Helm's codebase to enable greater capacity for change for Helm in the future.
+
+
+To provide transparency into Helm 4 changes, non-trivial functional changes to Helm 4 are to be managed with formal proposal.
+A proposal can either be a standard HIP, or a lighter weight Helm 4 change proposal (similar to Helm 3 proposals: [hips/archives/helm/helm-v3](https://github.com/helm/community/tree/main/hips/archives/helm/helm-v3)). Details follow in the "TBD" section.
+
+
+It is desired to enable the wider Helm community to contribute changes easily and efficently.
+In order to coordinate changes, each significant change or feature will require a sponser who get approval for a Helm 4 proposal or HIP, and work with the release engineer to get changes merged.
+This elease engineer and managed b
+-->
 
 ## Rationale
 
 During the Helm 3 process, Helm core maintainers received criticism that the development process was not described publicly enough.
+So this proposal attempts to spell out a formal process for a sucessful Helm 4 release.
+
 Further criticism was that it was unclear who was "running the show" and how they were making decisions.
 Finally, we were criticized for not explaining the criteria we were using for determining when Helm 3 was complete.
-This document is intended to address these criticisms.
-It describes the process in detail, providing both a method for us to follow and a point of reference for community members to read.
+
+This document is intended to address these criticisms and present a reasonable plan for timely delivery of a valuable Helm 4 release to the community.
+It describes the process in detail, providing both a method for us to follow and a point of reference for community members to follow for contributing changes.
+
+The process based on the following ideas:
+
+- Requiring changes to be documented (either Helm 4 proposals, or standard HIPs).
+Both allows Helm communitity members to review changes, and allow Helm maintainers to provide oversight on the functional and architectual implementation of the given change. Ensuring changes are in good standing with respect to this HIP.
+Finally, documented changes should allow contributors to independently execute changes to the agreed proposal in coordination with the release engineer.
+
+- Requiring a timeboxed development effort intends to both promote focus on the most valuable changes that the community/maintainers can deliver within a reasonable timeframe.
+And avoid an "open ended" development cycle that might drag on and/or fizzle out.
+Ideally the majority of changes will be proposed and agree upon early in the development cycle (the planning phase).
+But changes to this initial scope can be ammended during the course of the development lifecycle.
+So long as they can fit within the agreed upon timeframe.
+Making amending the scope an exlicit requirement seeks to ensure that any changes are well thought out.
+For further changes that don't make it into Helm 4, there will be a Helm 5 of course!
+
 
 ## Specification
 
+### Prerequisites
+Before Helm can start the development of Helm v4 there are some things the Helm organization needs to have in order.
+These include:
+
+- Active maintainers who will take the time to review changes.
+Helm has been in an unoffical "maintenance mode" for some time where changes have slowed down and the project needs the people capable of driving a Helm 4.
+- Direction on what is in and out of scope for Helm 4.
+It’s important to be able to know what to say “yes” to and what to say “no” to with reasons for both.
+- A timeline for the development of Helm 4.
+A timeline sets expectations for the time of development.
+If the time for development is open-ended then development could end up in a perl 6 development cycle and just take forever.
+At some point development needs to stop.
+
+This HIP aims to define sucess criteria for these.
+
+### The process / timelines
+
+Helm 4 development will consist of three main phases:
+1. Planning
+1. Feature development
+1. Release preparation
+
+The timeline will be approximately:
+
+1. Marking this HIP as non-draft
+1. Naming of a Release Engineer: Nov. 2024
+1. Kick-off meeting: KubeCon/CloudNativeCon North America 2024 (Salt Lake City)
+1. Earliest that engineering can begin: Nov. 2024
+1. Expected release of Helm v4.0.0 will be planned for the week before KubeCon/CloudNativeCon North America 2025 (expected November 2025)
+
+Prior to the stable release there will be 4-6 weeks of release candidates for the public to test. The time period will be determined by the testing and state of trust in the code leading up to the release candidate window.
+
+Helm v4.1.0 will be in January and from then on it will follow the same release cadence that Helm v3 has, shadowing Kubernetes by approximately 1 month.
+
+Helm 3 will reach end of life six months after Helm 4 release (July. 2026 estimated)
+
+<!--
+The following timeline provides for a time boxed period of Helm v4 development.
+
+Development will open for Helm v4 at the start of November 2024
+Helm v4.1.0 will be in January and from then on it will follow the same release cadence that Helm v3 has, shadowing Kubernetes by approximately 1 month.
+
+-->
+
+### Change suitability
+
+The Helm ecosystem is large and encompasses not just the Helm CLI, but also the SDK, Charts, and Chart repositories, plugins, etc.
+Helm also has several [roles or personas](https://github.com/helm/community/blob/main/user-profiles.md) for different users categories
+From "Application Operators" to "Helm Developers".
+The Helm project must be judicious in how it delivers changes without significant adverse impact to Helm's users or the overall Helm ecosystem.
+
+So, while Helm 4 doesn't need not be semantically backwards compatible with Helm 3.
+It must not cause significant breakage for users choosing to adopt Helm 4.
+With migration options and paths clearly present and documented for Helm 3 users to update to Helm 4.
+
+### Change proposals
+
+In order to ensure Helm 4 changes are aligned to this HIP, Helm core mantainers will review proposals. Either in the form of Helm 4 specific proposals (intended to be a lighter-weight process to HIPs). Or HIPs, which deemed to be suitable for Helm 4 implementation.
+
+When a Helm 4 proposal is deemed suitable, it will be marked as implementable in the hips/helm-v4 [hips/helm/helm-v4](https://github.com/helm/community/tree/main/hips/helm/helm-v4) directory.
+Standard Helm HIPs, marked as suitable for Helm 4, will also be accepted.
+
+#### Helm 4 proposals
+
+"Helm 4 proposal" are simply a lightweight version of the Helm HIP process.
+Intended to reduce the friction to implement small changes for Helm 4.
+That don't warrent/require a full HIP process.
+See appendix for a template of the proposal.
+
+
+### Compatbility
+
+Even with Helm 4 being a major version release which does not need to adhere to [HIP-0004](./hip-0004.md) compatibility guarentees.
+As Helm is mature software, with a significant ecosystem built around the CLI, charts, the SDK, etc.
+Helm 4 must remain largely feature/functionality compatible with Helm 3.
+
+Where non-compatible changes are introduced, they must either have migration documentation for equivalent functionality.
+Removal of functionality should prefer deprecation for later removal.
+Behavioural changes that negatively affect a significant number of users (without a reasonable migration) can't be allowed.
+
+Specific requirements for compatibility include the below:
+
+#### Compatbility with existing charts and releases
+
+While the tooling can evolve, Helm must maintain compatibility with existing charts and releases.
+A Helm 4 which is unable to operate on Helm 3 charts (and vice versa) is practically a different tool.
+And as such would diverge the Helm ecosystem.
+
+Specifically, charts that can be deployed with Helm 3 must be deployable by Helm 4.
+And similarly, any release created by Helm 3 much be managable by Helm 4.
+Helm 3 must also be able to upgrade releases created by Helm 4.0 (Later releases in the Helm 4.x series may make Helm 3 incompatible changes)
+
+Exceptions for already deprecated functionality e.g. `requirements.yaml` may be made.
+
+#### Compatbility with existing CLI workflows
+
+The Helm CLI is ingrained in thousands of release pipelines and automation systems.
+So while the CLI can evolve, it must remain loosely compatible with existing users workflows.
+
+A judgement call will need to be made by the release engineer and maintainers to how much a given change will overall affect user workflows. And the options to mitigate those changes.
+
+As a baseline, it is desirable to the majority of Helm 3 CLI "Application Operator" users to be unaffected by Helm 4 upgrade.
+Or only need to make minor changes to their workflow to incorporate Helm 4 into their workflows e.g. adjusting CLI flags.
+With changes being clearly described in a migration document.
+
+Other users e.g. "Application Developers" tend to be more advanced. And can have more significant changes imposed at the judgement of the Release Engineer and maintainers.
+
+###
+
 ### The Release Engineer
 
 A single individual will oversee the planning and development phases of Helm 4.
@@ -44,11 +231,11 @@ The person will have the following responsibilities:
 - Determine the timelines for development
 - Make the determination of when the project has indeed reached a milestone
 - Name and oversee releases
-- After the release, the release manager will determine what the "intent" of a Helm behavior was (which informs determining whether an incoming breaking change is a feature or a bug fix)
+- After the release, the release engineer will determine what the "intent" of a Helm behavior was (which informs determining whether an incoming breaking change is a feature or a bug fix)
 
 #### Determining Who Will Be Release Engineer
 
-The release engineer must be a core maintainer on the `github.com/helm/helm` project.
+The release engineer must be a core maintainer on the <https://github.com/helm/helm> project.
 The engineer must not be in danger of losing core maintainer-ship due to inactivity.
 Anyone can nominate a core maintainer to become the release engineer (self-nominations are encouraged).
 Any nominee must _agree_ to the nomination.
@@ -57,23 +244,6 @@ The Helm project maintainers will determine which of the nominated maintainers s
 In the event that a release engineer cannot perform the duties required,
 the Helm project maintainers may choose a replacement.
 
-### The Process
-
-There will be three phases to Helm 4 development:
-
-1. Planning
-2. Feature development
-3. Release preparation
-
-The timeline will be approximately:
-
-- Naming of a Release Engineer: Sept. 2021
-- Kick-off meeting: Oct. 2021
-- Earliest that engineering can begin: Jan. 2022
-- Earliest that a release could possibly be: May, 2022
-
-More likely, the release will be late 2022 to midyear 2023.
-
 #### Planning
 
 This phase begins when the release engineer is named.
@@ -116,7 +286,7 @@ This phase will terminate, and Helm 4.0.0 development will be deemed complete, u
 
 However, this phase must not be less than three months in order to give the broader ecosystem ample time to test and provide feedback.
 
-#### Halting Helm 4 Development
+### Halting Helm 4 Development
 
 During the course of development, it may turn out that there is not sufficient reason to continue development on Helm 4.
 For example, there may not be enough feature HIPs to justify releasing,
@@ -126,26 +296,28 @@ In this case, the Helm maintainers may hold a "circuit breaker" vote to halt Hel
 
 The process for halting Helm 4 development will be a simple majority vote by the Helm project maintainers.
 
-#### Maintaining Helm 3
+### Maintaining Helm 3
 
 This section outlines how we will support Helm 3 during Helm 4 development. However, the Release Engineer is at liberty to refine this section in the name of maintaining the best experience for Helm users.
 
 During the "Feature Development" cycle of Helm 4, Helm 3 will continue to receive security, bug, and minor feature patches.
 
 During the Alpha, Beta, and RC phases of Helm 4, Helm 3 will not receive any feature patches. This is to prevent Helm 3 from introducing features that should be in Helm 4, but during a window when Helm 4's features are frozen.
 
-Helm 3 will receive 6 months of bug fixes and one year of security fixes from the day that Helm 4 is released.
+Helm 3 will receive 6-8 months of bug fixes and one year of security fixes from the day that Helm 4 is released.
 
 ## Backwards compatibility
 
 Nothing in this document is contrary to our existing process for decision making.
+Specifically [HIP-0004](./hip-0004.md) largely applies to minor/patch releases only.
+And not a major version release.
 Note that this does not change the PR review process or the HIP process.
 It assumes that both of those processes will remain the same.
 
 ## How to teach this
 
 The contents of this HIP will be discussed in the public Helm Dev meeting, which is recorded.
-If possible, the kick-off meeting will be held at KubeCon North America in 2021,
+If possible, the kick-off meeting will be held at KubeCon North America in 2024,
 and the resources of CNCF will be utilized for advertising this meeting and its purposes.
 
 During all stages of Helm 4 development, the Release Engineer will ensure that an update on progress is given at no fewer than two Helm public dev calls each month.
@@ -156,8 +328,8 @@ Helm 3 was largely developed along an informal version of the above model.
 However, we were criticized for not documenting the process or explaining how we were going to follow the process.
 This proposal provides an alternative to an "informal process" that would draw similar ire.
 
-Multiple release managers was also considered.
-However, two things suggest multiple release managers would not be appropriate:
+Multiple release engineers were also considered.
+However, two things suggest multiple release engineers would not be appropriate:
 
 1. The desire to have a single authority for architectural integrity and resolution
 2. The simple fact that there are not enough Helm core maintainers to justify having a committee
@@ -193,4 +365,12 @@ It is in the community's best interest for the developers to move at a deliberat
 ## References
 
 - The Helm org's [governance documents](https://github.com/helm/community/tree/master/governance)
-- The [Helm 3 planning documents](https://github.com/helm/community/tree/master/hips/archives/helm/helm-v3)
+- The [Helm 3 planning documents](https://github.com/helm/community/tree/master/hips/archives/helm/helm-v3)
+
+## Appendix
+
+### Helm 4 proposal template
+
+```
+TBD
+```