knative-operator removes the istio inject label from knative-serving namespace

[jrhunger]

Is anyone else seeing this? Have i missed a config option that controls this?
I followed this guide to install the operator: https://knative.dev/docs/install/operator/knative-with-operators/
The only change i made is to download the manifest and edit it throughout to change the namespace from default to knative-operator.

Here is my knative-serving.yaml:

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  config:
    domain:
      "mydomain.mycompany.com": ""
    istio:
      local-gateway.knative-serving.knative-local-gateway: "knative-local-gateway.istio-ingress.svc.cluster.local"
  ingress:
    istio:
      enabled: true
      knative-ingress-gateway:
        selector:
          istio: ingress
      knative-local-gateway:
        selector:
          istio: ingress

I kept getting "no healthy upstream" until i enabled istio injection on the knative-serving namespace. Everything worked then for a while and abruptly started returning "no healthy upstream". I found the istio injection annotation had been removed from knative-serving namespace and in my apiserver logs i see that the knative-operator did it:

			"auditID": "d42c78c5-2953-476c-8002-2bcd34883013",
			"requestReceivedTimestamp": "2024-03-27T04:15:37.364080Z",
			"objectRef": {
				"uid": "8d45d8c3-793e-4e59-b9fe-9245f5bdf771",
				"apiVersion": "v1",
				"resource": "namespaces",
				"resourceVersion": "14886470",
				"namespace": "knative-serving",
				"name": "knative-serving"
			},
			"level": "Metadata",
			"kind": "Event",
			"verb": "update",
			"annotations": {
				"authorization": {
					"k8s": {
						"io/reason": "RBAC: allowed by ClusterRoleBinding \"knative-eventing-operator-aggregated-stable\" of ClusterRole \"knative-eventing-operator-aggregated-stable\" to ServiceAccount \"knative-operator/knative-operator\"",
						"io/decision": "allow"
					}
				},
				"mutation": {
					"webhook": {
						"admission": {
							"k8s": {
								"io/round_0_index_3": "{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"
							}
						}
					}
				}
			},
			"userAgent": "operator/v0.0.0 (linux/amd64) kubernetes/$Format",
			"requestURI": "/api/v1/namespaces/knative-serving?fieldManager=manifestival",
			"responseStatus": {
				"code": 200
			},
			"stageTimestamp": "2024-03-27T04:15:37.379862Z",
			"sourceIPs": [
				"10.75.144.128"
			],
			"apiVersion": "audit.k8s.io/v1",
			"stage": "ResponseComplete",
			"user": {
				"uid": "de40d58f-e710-4aca-b5a4-78f7b58d7c2a",
				"extra": {
					"authentication": {
						"kubernetes": {
							"io/pod-uid": [
								"02328dc1-ff93-446c-8758-4bafe9def62b"
							],
							"io/pod-name": [
								"knative-operator-8556df77d5-vf69c"
							]
						}
					}
				},
				"groups": [
					"system:serviceaccounts",
					"system:serviceaccounts:knative-operator",
					"system:authenticated"
				],
				"username": "system:serviceaccount:knative-operator:knative-operator"
			}

[jrhunger]

Looks like there is an issue for this in knative/operator, along with an unsatisfactory workaround in the issue discussions. knative/operator#388

[dprotaso]

Let's move the discussion there - thanks for finding the issue