Host receiving constant inserts will never succeed in the `wait for query finish` check

[tanner-bruce]

For observability/telemetry data if a host is constantly receiving inserts it ends up always waiting for the full wait for query finish interval to rollout and restart.

We use a service template for ingest-<cluster>, and insert directly to these pods (no distributed insert). Feature request would be to allow removing these pods from this service template before waiting for queries to finish. So that we do not have to interrupt the inserts, but also gives us the control to shut them down a bit quicker

[alex-zaitsev]

@tanner-bruce , you can disable wait for queries globally or on a CHI level.

In operator configuration:

spec:
  reconcile:
    host:
      wait:
        queries: "false"

In CHI:

spec:
  reconciling:
    policy: nowait

[tanner-bruce]

@alex-zaitsev my understanding was that this will immediately restart the pod. I would prefer to have currently running inserts finish since they can be relatively large. I think setting ready: false before waiting would accomplish this

[alex-zaitsev]

Could you share service template for the service you are using for inserts?

[tanner-bruce]

This is set as the clusterServiceTemplate

    - generateName: chi-{cluster}-cluster
      metadata:
        annotations:
          cloud.google.com/load-balancer-type: Internal
          networking.gke.io/internal-load-balancer-allow-global-access: "true"
      name: core-cluster-service-template
      spec:
        ports:
        - name: http
          port: 8123
          targetPort: 8123
        - name: tcp
          port: 9000
          targetPort: 9000
        - name: https
          port: 8443
          protocol: TCP
          targetPort: 8443
        - name: tcp-tls
          port: 9440
          protocol: TCP
          targetPort: 9440
        type: LoadBalancer