Proposal: Replace ConfigMap-based deployment configuration with Custom Resource

[FogDong]

Is your feature request related to a problem? Please describe.
When testing llm-d, I encountered several issues with the current ConfigMap-based deployment configuration:

• The configuration is hard to read and modify due to the ConfigMap merging behavior
• Changes to ConfigMap don't automatically trigger deployment updates

In my testing environment with node taints, I had to:

1. Manually modify the ConfigMap to add tolerations
2. Delete the deployment to force the controller to reconcile
3. Wait for the new deployment with updated configuration

Describe the solution approach you'd like
Replace ConfigMap with a dedicated Custom Resource for deployment configuration. Example:

apiVersion: llmd.ai/v1alpha1
kind: ModelServiceConfig
metadata:
  name: cfg
spec:
  modelId: gpt-35-turbo
  prefillDeployment: 
      replicas: 2,
      resources: 
          requests:
            cpu: "100m"
            memory: "128Mi"
      tolerations:
          - key: "nvidia.com/gpu"
            operator: "Exists"
            effect: "NoSchedule"

This approach would:

• Provide better type safety and validation
• Enable automatic reconciliation on changes with watching this CR
• Improve configuration visibility and management
• Allow direct JSON configuration through *runtime.RawExtension or just deployment struct

Describe alternatives you've considered

Additional context
The current ConfigMap approach makes it particularly challenging in environments with special requirements (like node taints or specific resource constraints). A Custom Resource would provide a more native Kubernetes experience and better integration with the operator pattern.

[feloy]

If workforce is needed, I would be happy to work on the implementation of this proposal. Please let me know, and also which public meeting is concerned to talk about this

[asm582]

Thanks for opening this issue. We agree that CR would be a good step forward. In the spirit of collecting more use cases, can you please share why adding tolerations to msvc does not work for your use case?

[FogDong]

Thanks for opening this issue. We agree that CR would be a good step forward. In the spirit of collecting more use cases, can you please share why adding tolerations to msvc does not work for your use case?

According to the msvc CRD, there's no tolerations field in the spec.