OCPBUGS-41903: operator/status: clear azure path fix job conditions on operator removal #1142
Conversation
there is sometimes a race condition when setting .spec.managementState back and forth between Removed and Managed where the azure path fix controller will kick off the job, but the resources needed for the job to run will get removed (as expected) before the job can finish and its controller update the operator's progressing condition to reflect that. this has been happening often in the TestLeaderElection e2e test. this test does not wait for the image registry operator to become available before removing it, and this is what triggers the race. a customer has reported a similar issue, although their error was slightly different and proved harder to reproduce. this commit should fix both problems. clearing the status is most important when users upgrade from a version of the azure path fix controller that deploys the job. OCP versions that do not deploy the job should not have the problem, and including this code in them should be harmless. in OCP >= 4.17 this job is no longer deployed, though the controller is kept it can probably be safely removed in OCP >= 4.18.
openshift-ci-robot
added
jira/severity-critical
|
@flavianmissi: This pull request references Jira Issue OCPBUGS-41903, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (xiuwang+1@redhat.com), skipping review request. The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: flavianmissi The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/test e2e-azure-operator |
|
@flavianmissi: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
will re-run
|
|
/label qe-approved |
openshift-ci
bot
added
qe-approved
|
@flavianmissi: This pull request references Jira Issue OCPBUGS-41903, which is valid. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (xiuwang+1@redhat.com), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
This seem to behave well overall but I'm still slightly skeptical :D |
|
@flavianmissi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/04ea1110-8d30-11ef-8a54-0b936220e312-0 |
|
/payload-abort |
|
@flavianmissi: aborted active payload jobs for pull request #1142 |
|
/payload-aggregate periodic-ci-openshift-release-master-ci-4.18-upgrade-from-stable-4.17-e2e-azure-ovn-upgrade 5 |
|
@flavianmissi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6d3e5ee0-8d32-11ef-80b2-c38a9eac7d34-0 |
|
/payload-abort |
|
@flavianmissi: aborted active payload jobs for pull request #1142 |
flavianmissi
force-pushed
the
OCPBUGS-41903
branch
from
4ca6c94
to
163815b
Compare
|
Test failures are infra related - I'll allow CI some time to recover before retesting. |
|
/retest |
|
@flavianmissi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/71599d90-8d60-11ef-9619-1d5ed6ce6387-0 |
|
/retest |
there is sometimes a race condition when setting .spec.managementState back and forth between Removed and Managed where the azure path fix controller will kick off the job, but the resources needed for the job to run will get removed (as expected) before the job can finish and its controller update the operator's progressing condition to reflect that.
this has been happening often in the TestLeaderElection e2e test. this test does not wait for the image registry operator to become available before removing it, and this is what triggers the race. a customer has reported a similar issue, although their error was slightly different and proved harder to reproduce. this commit should fix both problems.
clearing the status is most important when users upgrade from a version of the azure path fix controller that deploys the job. OCP versions that do not deploy the job should not have the problem, and including this code in them should be harmless.
in OCP >= 4.17 this job is no longer deployed, though the controller is kept it can probably be safely removed in OCP >= 4.18.
Testing
We need upgrade tests as well as regression tests for this one. Testing operator removal (
.spec.managementState: Removed) would also be great.