Install remote Knative components in canonical namespaces

[kahirokunn]

Fixes #2345

Proposed Changes

• Install remote KnativeServing and KnativeEventing components in their canonical spoke namespaces regardless of the hub management CR namespace.

Design

The component kind determines the remote installation namespace:

Management CR on the hub	Installation namespace on the spoke
KnativeServing	knative-serving
KnativeEventing	knative-eventing

Local installations without spec.clusterProfileRef continue to use the management CR metadata.namespace.

Verification Steps

E2E verification with Kind

Verified manually with three Kind clusters (hub, cluster1, and cluster2), OCM-managed ClusterProfile objects, the managed-serviceaccount credential syncer, cp-creds, and Envoy Gateway.

1. Set up OCM and ClusterProfile access

OCM was initialized on hub with ClusterProfile support. cluster1 and cluster2 were joined, accepted, assigned to the sandbox-fleet ManagedClusterSet, and bound to the knative-operator namespace. Both managed clusters reported Joined=True and Available=True; both ClusterProfiles reported ControlPlaneHealthy=True and Joined=True.

The managed-serviceaccount credential syncer and the cluster-proxy user server were enabled:

helm upgrade --install managed-serviceaccount ocm/managed-serviceaccount \
  --kube-context kind-hub \
  -n open-cluster-management-managed-serviceaccount \
  --create-namespace \
  --version 0.10.0 \
  --set featureGates.clusterProfileCredSyncer=true

helm upgrade --install cluster-proxy ocm/cluster-proxy \
  --kube-context kind-hub \
  -n open-cluster-management-cluster-proxy \
  --create-namespace \
  --version 0.11.0 \
  --set featureGates.clusterProfile=true \
  --set userServer.enabled=true \
  --set enableServiceProxy=true

The knative-operator ManagedServiceAccount was created for each spoke and labeled for ClusterProfile credential synchronization. The corresponding spoke service accounts were bound to cluster-admin for this installation test. The resulting cluster1-knative-operator and cluster2-knative-operator credentials appeared in the hub's knative-operator namespace, and both ClusterProfiles advertised the open-cluster-management access provider.

2. Configure the operator to use cp-creds

The operator mounted quay.io/open-cluster-management/cp-creds:latest at /access-plugins/cp-creds and used this provider configuration:

{
  "providers": [
    {
      "name": "open-cluster-management",
      "execConfig": {
        "apiVersion": "client.authentication.k8s.io/v1",
        "command": "/access-plugins/cp-creds/cp-creds",
        "args": ["--managed-serviceaccount=knative-operator"],
        "provideClusterInfo": true,
        "interactiveMode": "Never"
      }
    }
  ]
}

3. Build and deploy this PR head

PATH="$(go env GOPATH)/bin:$PATH" \
  KO_DOCKER_REPO=kind.local \
  KIND_CLUSTER_NAME=hub \
  ko apply -f config/

kubectl --context kind-hub -n knative-operator \
  rollout status deployment/knative-operator --timeout=180s

Both referenced ClusterProfiles resolved through the configured cp-creds provider.

4. Configure the Serving data plane on cluster1

Gateway API v1.4.1 experimental CRDs and Envoy Gateway v1.7.1 were installed on cluster1 with the GatewayNamespace deployment model. The following data-plane resources were configured:

Traffic	Gateway namespace/name	Envoy Service
External	eg-external/eg-external	eg-external/knative-external
Cluster-local	eg-internal/eg-internal	eg-internal/knative-internal

Both GatewayClasses reported Accepted=True; the internal Gateway reported Programmed=True. Because this local environment did not expose a LoadBalancer address, external traffic was verified through the external Envoy Service's NodePort on the Kind Docker network.

5. Deploy Serving and Eventing from non-canonical hub namespaces

The management CR namespaces intentionally differed from the remote installation namespaces:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: ocm-serving
  namespace: management-serving-hub
spec:
  clusterProfileRef:
    name: cluster1
    namespace: knative-operator
  ingress:
    gateway-api:
      enabled: true
    istio:
      enabled: false
---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
metadata:
  name: ocm-eventing
  namespace: management-eventing-hub
spec:
  clusterProfileRef:
    name: cluster2
    namespace: knative-operator

Both hub CRs reached TargetClusterResolved=True and Ready=True. All seven Serving Deployments became Available only in cluster1/knative-serving; Knative Eventing reached Ready=True with its controller, webhook, and broker Deployments only in cluster2/knative-eventing. No Knative Deployments appeared in either hub management namespace or on the wrong spoke.

The remote anchors were created as knativeserving-ocm-serving-root-owner and knativeeventing-ocm-eventing-root-owner, and managed resources referenced the matching anchor.

6. Verify Serving traffic

kubectl --context kind-cluster1 apply -f - <<'EOF'
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
      - image: gcr.io/knative-samples/helloworld-go
        env:
        - name: TARGET
          value: OCM canonical namespace verification
EOF

kubectl --context kind-cluster1 -n default \
  wait --for=condition=Ready kservice/helloworld-go --timeout=300s

NODE_PORT=$(kubectl --context kind-cluster1 -n eg-external \
  get service knative-external \
  -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')

docker run --rm --network kind curlimages/curl:8.15.0 \
  -H 'Host: helloworld-go.default.example.com' \
  "http://cluster1-control-plane:${NODE_PORT}"

Observed response:

Hello OCM canonical namespace verification!

The external and cluster-local HTTPRoutes both reported Accepted=True and ResolvedRefs=True.

7. Verify cleanup and recreation

Both management CRs were deleted:

kubectl --context kind-hub -n management-serving-hub \
  delete knativeserving ocm-serving
kubectl --context kind-hub -n management-eventing-hub \
  delete knativeeventing ocm-eventing

Both CRs completed finalization. Their anchors and all anchor-owned Deployments were removed from cluster1/knative-serving and cluster2/knative-eventing. After recreating the same management CRs, both returned to TargetClusterResolved=True and Ready=True, and the Serving request returned the same response again.

Upgrade note

Remote installations created before this change used the management CR namespace on the spoke. If an existing remote installation is not already in its canonical namespace:

1. Delete the management CR with the old operator and wait for finalization to remove the spoke resources.
2. Upgrade the operator.
3. Recreate the same management CR on the hub.

The recreated installation is placed in knative-serving or knative-eventing. No migration is required when the existing remote installation already uses its canonical namespace.

Release Note

action required: Remote KnativeServing and KnativeEventing installations now use the canonical knative-serving and knative-eventing namespaces instead of the management CR namespace. Delete non-canonical remote installations with the old operator before upgrading, then recreate their management CRs.

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kahirokunn
Once this PR has been reviewed and has the lgtm label, please assign arsenetar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.95%. Comparing base (71eefe9) to head (7ce0e6f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2349      +/-   ##
==========================================
+ Coverage   65.84%   65.95%   +0.11%     
==========================================
  Files          56       56              
  Lines        2673     2682       +9     
==========================================
+ Hits         1760     1769       +9     
  Misses        777      777              
  Partials      136      136              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kahirokunn]

I plan to implement a separate validation rule to prevent the creation of more than one KnativeServing or KnativeEventing instance per cluster.

[kahirokunn]

@dsimansk Hi ✋ PTAL. Thank you 🙏