feat: operator watch namespaces

[RobertSamoilescu]

What this PR does / why we need it:

This PR implements the option to configure the operator to watch specific namespaces.

To install the operator to watch specific namespaces, run:

helm upgrade seldon-core-v2-setup ./seldon-core-v2-setup/   --namespace seldon-mesh   --set "controller.watchNamespaces={seldon-mesh,seldon-mesh-2}"   --set controller.clusterwide=false   --install

In the example above, the operator will watch only seldon-mesh and seldon-mesh-2 namespaces.

Configuration details:

• if clusterwide flag is set to true and watchNamespace is not set, then all the namespaces will be watched
• if clusterwide flag is set to true and watchNamespaces is set, then the operator will watch the namespaces specified
• if clusterwide flag is set to false, then the operator will watch the POD_NAMESPACE which is the same as the release namespace.

Additionally, if a user wants to have a controller in ns1 watching the seldon-mesh-1, and another controller in ns2 watching seldon-mesh-2, they can use --set controller.skipClusterRoleCreation to overcome the error from recreating the same cluster role twice.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

[lc525]

There is an additional situation we will want to cover, but we might choose to reprioritise and deal with it in a separate PR:

• clusterwide: True , watchNamespaces not set -- watch all namespaces
• clusterwide: True, watchNamespaces is set and we allow the operator ClusterRole/ClusterRoleBinding permissions. Here, the operator only watches some namespaces but has the permissions to watch any namespace. In security-restricted environments, this is problematic. However, other operators like postgres or spark also only offer this option.
• clusterwide: False, watchNamespaces not set -- only watch the namespace where the operator is installed
• clusterwide: False, watchNamespaces is set -- we only give permissions to the operator to watch the specific list of namespaces. The helm chart needs to create RoleBindings in each namespace in the list (or which matches a specific criteria) during the call to helm install or helm upgrade. These RoleBindings should give the operator’s service account the necessary privileges in the namespace.

[lc525]

Before we merge this, let's test for sane behaviour when instantiating two operators in ns1 and ns2, both watching the same namespace seldon-mesh. We only want one of them to do the actual management / connect to the scheduler and issue control-plane commands.

[RobertSamoilescu]

Before we merge this, let's test for sane behaviour when instantiating two operators in ns1 and ns2, both watching the same namespace seldon-mesh. We only want one of them to do the actual management / connect to the scheduler and issue control-plane commands.

In this case, although the scheduler seems to behave ok for model loading/unloading, there are some errors which pop up in the operator.

When loading:

2025-05-01T14:10:13.851Z    ERROR    Reconciler error    {"controller": "model", "controllerGroup": "mlops.seldon.io", "controllerKind": "Model", "Model": {"name":"iris","namespac │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler                                                                                          │
│     sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:341                                                                                                │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem                                                                                       │
│     sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:288                                                                                                │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2                                                                                             │
│     sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:249  

When unloading:

2025-05-01T14:11:15.778Z    ERROR    schedulerClient.SubscribeModelEvents    Failed to update status    {"model": "iris", "error": "Model.mlops.seldon.io \"iris\" not found"}      │
│ github.com/seldonio/seldon-core/operator/v2/scheduler.(*SchedulerClient).SubscribeModelEvents                                                                                       │
│     github.com/seldonio/seldon-core/operator/v2/scheduler/model.go:270                                                                                                              │
│ github.com/seldonio/seldon-core/operator/v2/scheduler.retryFn.func2                                                                                                                 │
│     github.com/seldonio/seldon-core/operator/v2/scheduler/client.go:327                                                                                                             │
│ github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1                                                                                                   │
│     github.com/cenkalti/backoff/[email protected]/retry.go:18                                                                                                                               │
│ github.com/cenkalti/backoff/v4.doRetryNotify[...]                                                                                                                                   │
│     github.com/cenkalti/backoff/[email protected]/retry.go:88                                                                                                                               │
│ github.com/cenkalti/backoff/v4.RetryNotifyWithTimer                                                                                                                                 │
│     github.com/cenkalti/backoff/[email protected]/retry.go:61                                                                                                                               │
│ github.com/cenkalti/backoff/v4.RetryNotify                                                                                                                                          │
│     github.com/cenkalti/backoff/[email protected]/retry.go:49                                                                                                                               │
│ github.com/seldonio/seldon-core/operator/v2/scheduler.retryFn                                                                                                                       │
│     github.com/seldonio/seldon-core/operator/v2/scheduler/client.go:329                                                                                                             │
│ github.com/seldonio/seldon-core/operator/v2/scheduler.(*SchedulerClient).startEventHanders.func1                                                                                    │
│     github.com/seldonio/seldon-core/operator/v2/scheduler/client.go:95    

This is a faulty configuration anyways, and maybe we should address it in a future PR.

[lc525]

This is the cluster role specific to the operator. Perhaps we should reflect that in the value name? Something like skipOperatorClusterRoleCreation

[lc525]

lgtm, just a small suggestion on the helm value name