Skip to content

fix: convert one-time jobs to Helm hooks to prevent ArgoCD OutOfSync #899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pinkneyj wants to merge 5 commits into
base: develop
Choose a base branch
from
+18 −23
Open

fix: convert one-time jobs to Helm hooks to prevent ArgoCD OutOfSync #899

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fffe770
update affinity on server and aggregator to be automatically set
pinkneyj Jul 3, 2026
d32537a
add helm hook/delete to backfill,config loader, helmless and init cer…
pinkneyj Jul 8, 2026
41bcd83
minor yaml changes
pinkneyj Jul 8, 2026
b76d609
ensure other changes no included in this PR
pinkneyj Jul 8, 2026
ee17e79
remove checksum naming
pinkneyj Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions helm/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -786,7 +786,7 @@ Map for initBackfillJob values; this allows us to preferably use initBackfillJob
Name for a job resource
*/}}
{{- define "cloudzero-agent.jobName" -}}
{{- printf "%s-%s-%s" .Release .Name (include "cloudzero-agent.configurationChecksum" .) | trunc 61 -}}
{{- printf "%s-%s" .Release .Name | trunc 63 -}}
{{- end }}

{{/*
Expand Down Expand Up @@ -818,7 +818,7 @@ from the checksum.
Name for the backfill job resource
*/}}
{{- define "cloudzero-agent.initBackfillJobName" -}}
{{- printf "%s-backfill-%s" .Release.Name (include "cloudzero-agent.configurationChecksum" .) | trunc 52 -}}
{{- printf "%s-backfill" .Release.Name | trunc 63 -}}
{{- end }}

{{/*
Expand All @@ -839,14 +839,14 @@ annotations:
{{- end -}}

{{/*
Name for the certificate init job resource. Should be a new name each installation/upgrade.
Name for the certificate init job resource.
*/}}
{{- define "cloudzero-agent.initCertJobName" -}}
{{- include "cloudzero-agent.jobName" (dict "Release" .Release.Name "Name" "init-cert" "Version" .Chart.Version "Values" .Values) -}}
{{- end }}

{{/*
Name for the helmless job resource. Should be a new name each installation/upgrade.
Name for the helmless job resource.
*/}}
{{- define "cloudzero-agent.helmlessJobName" -}}
{{- include "cloudzero-agent.jobName" (dict "Release" .Release.Name "Name" "helmless" "Version" .Chart.Version "Values" .Values) -}}
Expand Down
25 changes: 7 additions & 18 deletions helm/templates/backfill-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,31 +16,19 @@
immediately upon helm install to capture existing resources without waiting
for the first scheduled run.

Helm hooks (post-install, hook-delete-policy) were tried, but rejected because
they require Helm-specific features that don't work with helm template,
kubectl apply, ArgoCD, Flux, or other GitOps tools
The one-time Job uses Helm hooks (pre-install, pre-upgrade) so that it is not
tracked as a managed resource by ArgoCD or Helm, preventing OutOfSync errors
when the job completes and is deleted. The CronJob is left as a standard
resource as it manages its own lifecycle.

Note: ArgoCD must be configured with --enable-helm-hooks for hooks to run.

The implementation uses a range loop with small if/else clauses to maximize
code reuse while handling the minimal differences between CronJob and Job
specifications.
*/ -}}
{{- range $jobType := list "CronJob" "Job" }}
{{- $jobCategory := ternary "cronjob" "onetime" (eq $jobType "CronJob") }}
{{- /*
Istio mTLS Port Exclusion

When Istio is detected and cert-manager is not in use, exclude port 443 from
Envoy proxying. The backfill job connects to the webhook service on port 443
(the service port, which Kubernetes routes to pod port 8443). Without this
exclusion, Istio wraps the connection in mTLS, but the webhook uses a
self-signed certificate that Istio doesn't recognize, causing TLS failures.

Port 443 is excluded (not 8443) because Istio's sidecar intercepts outbound
traffic based on the destination port the application connects to - the
service port, not the pod's target port.

See helm/docs/istio.md for full documentation.
*/ -}}
{{- $istioAnnotations := dict -}}
{{- if and (include "cloudzero-agent.Values.integrations.istio.enabled" $) (not $.Values.insightsController.tls.useCertManager) -}}
{{- $istioAnnotations = dict "traffic.sidecar.istio.io/excludeOutboundPorts" "443" -}}
Expand All @@ -57,6 +45,7 @@ metadata:
$.Values.initBackfillJob.annotations
$.Values.components.webhookServer.annotations
$.Values.components.webhookServer.backfill.annotations
(ternary (dict "helm.sh/hook" "pre-install,pre-upgrade" "helm.sh/hook-delete-policy" "before-hook-creation") (dict) (eq $jobType "Job"))
)
) | nindent 2 }}
{{- include "cloudzero-agent.generateLabels" (dict
Expand Down
2 changes: 2 additions & 0 deletions helm/templates/config-loader-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ metadata:
.Values.defaults.annotations
.Values.components.miscellaneous.configLoader.annotations
(dict "checksum/values" (include "cloudzero-agent.configurationChecksum" .))
(dict "helm.sh/hook" "pre-install,pre-upgrade")
(dict "helm.sh/hook-delete-policy" "before-hook-creation")
Comment on lines +12 to +13

@greptile-apps greptile-apps Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Pre-install hooks lack dependencies

On a fresh installation, Helm runs these pre-install Jobs before creating their ordinary ConfigMaps, service accounts, Secrets, and RBAC resources. The config-loader and helmless hooks are always rendered, so their pods cannot start and the installation blocks before the required release resources are created.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: helm/templates/config-loader-job.yaml
Line: 12-13

Comment:
**Pre-install hooks lack dependencies**

On a fresh installation, Helm runs these pre-install Jobs before creating their ordinary ConfigMaps, service accounts, Secrets, and RBAC resources. The config-loader and helmless hooks are always rendered, so their pods cannot start and the installation blocks before the required release resources are created.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code

Comment on lines +12 to +13

@greptile-apps greptile-apps Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Pre-upgrade hook reads stale configuration

When an upgrade changes values represented in the agent, validator, helmless, webhook, or aggregator ConfigMaps, this pre-upgrade hook runs before Helm applies the new ordinary manifests. It therefore loads and validates the previous revision's configuration, causing the deployed revision and the configuration sent to CloudZero to disagree until a later successful run.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: helm/templates/config-loader-job.yaml
Line: 12-13

Comment:
**Pre-upgrade hook reads stale configuration**

When an upgrade changes values represented in the agent, validator, helmless, webhook, or aggregator ConfigMaps, this pre-upgrade hook runs before Helm applies the new ordinary manifests. It therefore loads and validates the previous revision's configuration, causing the deployed revision and the configuration sent to CloudZero to disagree until a later successful run.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code

)
) | nindent 2 }}
{{- include "cloudzero-agent.generateLabels" (dict
Expand Down
2 changes: 2 additions & 0 deletions helm/templates/helmless-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ metadata:
"annotations" (list
.Values.defaults.annotations
.Values.components.miscellaneous.helmless.annotations
(dict "helm.sh/hook" "pre-install,pre-upgrade")
(dict "helm.sh/hook-delete-policy" "before-hook-creation")
)
) | nindent 2 }}
{{- include "cloudzero-agent.generateLabels" (dict
Expand Down
4 changes: 3 additions & 1 deletion helm/templates/init-cert-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ metadata:
.Values.defaults.annotations
.Values.initCertJob.annotations
.Values.components.miscellaneous.initCert.annotations
(dict "helm.sh/hook" "pre-install,pre-upgrade")
(dict "helm.sh/hook-delete-policy" "before-hook-creation")
)
) | nindent 2 }}
) | nindent 2 }}
{{- include "cloudzero-agent.generateLabels" (dict
"root" .
"name" "init-cert"
Expand Down