Grid Search stuck when parallelTrialCount < maxTrialCount

[sidpalas]

/kind bug

What steps did you take and what happened:

I created an experiment to perform a grid search, set maxTrialCount equal to the size of the grid, and set parallelTrialCount < maxTrialCount.

The first set of parallel trials completes successfully, but at some point after that the experiment gets stuck. The suggestion shows requested > assigned and has the following warning:
Warning ReconcileError 4s (x12 over 31s) suggestion-controller The response contains unexpected trials

What did you expect to happen:
Experiment to execute trials for each point in the parameter space and then complete.

Anything else you would like to add:

I am running this image for the katib-controller: gcr.io/kubeflow-images-public/katib/v1alpha3/katib-controller:latest (I also tested with gcr.io/kubeflow-images-public/katib/v1alpha3/katib-controller:v0.8.0)

The issue seems like it is likely related to #1494

If I set parallelTrialCount >= maxTrialCount (enabling all runs to execute immediately) the experiment succeeds.

Environment:

• Kubeflow version (kfctl version): v1.0.1
• Minikube version (minikube version): N/A
• Kubernetes version: (use kubectl version): v1.18.17-gke.100
• OS (e.g. from /etc/os-release):

---

experiment.yaml:

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: sid
  labels:
    controller-tools.k8s.io: "1.0"
  name: minimal-grid
spec:
  objective:
    type: maximize
    goal: 999999
    objectiveMetricName: value
  resumePolicy: Never
  algorithm:
    algorithmName: grid
  parallelTrialCount: 5
  maxTrialCount: 15
  maxFailedTrialCount: 3
  parameters:
    - name: --param_1
      parameterType: int
      feasibleSpace:
        min: "0"
        max: "15"
  trialTemplate:
    goTemplate:
      rawTemplate: |-
        apiVersion: batch/v1
        kind: Job
        metadata:
          name: {{.Trial}}
          namespace: {{.NameSpace}}
        spec:
          template:
            spec:
              containers:
              - name: {{.Trial}}
                image: <IMAGE>
                command:
                - "python3"
                - "return_param_value.py"
                {{- with .HyperParameters}}
                {{- range .}}
                - "{{.Name}}={{.Value}}"
                {{- end}}
                {{- end}}
              restartPolicy: Never

Result of kubectl describe suggestion <suggestion-name>:

Name:         minimal-grid
Namespace:    sid
Labels:       controller-tools.k8s.io=1.0
Annotations:  <none>
API Version:  kubeflow.org/v1alpha3
Kind:         Suggestion
Metadata:
  Creation Timestamp:  2021-05-18T05:58:39Z
  Generation:          9
  Managed Fields:
    API Version:  kubeflow.org/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:controller-tools.k8s.io:
        f:ownerReferences:
      f:spec:
        .:
        f:algorithmName:
        f:requests:
      f:status:
        .:
        f:conditions:
        f:startTime:
        f:suggestionCount:
        f:suggestions:
    Manager:    katib-controller
    Operation:  Update
    Time:       2021-05-18T05:58:56Z
  Owner References:
    API Version:           kubeflow.org/v1alpha3
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Experiment
    Name:                  minimal-grid
    UID:                   90e538b2-5858-45d2-af0e-38c0837d6a8d
  Resource Version:        69731263
  Self Link:               /apis/kubeflow.org/v1alpha3/namespaces/sid/suggestions/minimal-grid
  UID:                     3875d508-b3ee-4426-88a7-1b307c381249
Spec:
  Algorithm Name:  grid
  Requests:        13
Status:
  Conditions:
    Last Transition Time:  2021-05-18T05:58:39Z
    Last Update Time:      2021-05-18T05:58:39Z
    Message:               Suggestion is created
    Reason:                SuggestionCreated
    Status:                True
    Type:                  Created
    Last Transition Time:  2021-05-18T05:58:54Z
    Last Update Time:      2021-05-18T05:58:54Z
    Message:               Deployment is ready
    Reason:                DeploymentReady
    Status:                True
    Type:                  DeploymentReady
    Last Transition Time:  2021-05-18T05:58:55Z
    Last Update Time:      2021-05-18T05:58:55Z
    Message:               Suggestion is running
    Reason:                SuggestionRunning
    Status:                True
    Type:                  Running
  Start Time:              2021-05-18T05:58:39Z
  Suggestion Count:        8
  Suggestions:
    Name:  minimal-grid-dq75pd5j
    Parameter Assignments:
      Name:   --param_1
      Value:  0
    Name:     minimal-grid-jsl78k9b
    Parameter Assignments:
      Name:   --param_1
      Value:  1
    Name:     minimal-grid-nbhlss7f
    Parameter Assignments:
      Name:   --param_1
      Value:  2
    Name:     minimal-grid-p87q65dk
    Parameter Assignments:
      Name:   --param_1
      Value:  3
    Name:     minimal-grid-dt2s5jd5
    Parameter Assignments:
      Name:   --param_1
      Value:  4
    Name:     minimal-grid-jxtrdnk9
    Parameter Assignments:
      Name:   --param_1
      Value:  6
    Name:     minimal-grid-l2t2gpc8
    Parameter Assignments:
      Name:   --param_1
      Value:  9
    Name:     minimal-grid-4dpdlj2v
    Parameter Assignments:
      Name:   --param_1
      Value:  13
Events:
  Type     Reason          Age               From                   Message
  ----     ------          ----              ----                   -------
  Warning  ReconcileError  0s (x8 over 15s)  suggestion-controller  The response contains unexpected trials

[andreyvelich]

Thank you for creating this @sidpalas.
It might be this issue: #1494 (comment).
I would recommend to use the latest Katib release to debug this.
v1alpha3 was deprecated 1 year ago.

[sidpalas]

Thanks @andreyvelich -- are there any special considerations I need to make when upgrading to the newer versions of Katib (e.g. related to DB migrations)?

I know for Kubeflow Pipelines upgrading in-place wasn't supported for some versions.

[andreyvelich]

Unfortunately, we don't support in-place upgrade currently.
You should manually delete all v1alpha3 Katib resources and use make deploy to use the latest version.

[andreyvelich]

Before clean-up resources you should delete all Experiments:
kubectl delete experiment --all-namespaces --all

[sidpalas]

Thanks, @andreyvelich!

I deployed a new standalone Katib deployment in a separate cluster to test and was able to reproduce this result.

I used the commit corresponding to the latest release (https://github.com/kubeflow/katib/releases/tag/v0.11.0). This deployed katib-controller with image: docker.io/kubeflowkatib/katib-controller:v0.11.0

The updated experiment spec (to now be compatible with v1beta1) is:

apiVersion: "kubeflow.org/v1beta1"
kind: Experiment
metadata:
  namespace: kubeflow
  name: grid-example-7
spec:
  objective:
    type: maximize
    goal: 999999
    objectiveMetricName: value
  algorithm:
    algorithmName: grid
  parallelTrialCount: 3
  maxTrialCount: 15
  maxFailedTrialCount: 3
  parameters:
    - name: param_1
      parameterType: int
      feasibleSpace:
        min: "0"
        max: "15"
  trialTemplate:
    primaryContainerName: main-container
    trialParameters:
      - name: param_1
        description: value passed as input
        reference: param_1
    trialSpec:
      apiVersion: batch/v1
      kind: Job
      spec:
        template:
          spec:
            containers:
              - name: main-container
                image: <IMAGE>
                command:
                  - "python3"
                  - "return_param_value.py"
                  - "--param_1=${trialParameters.param_1}"
            restartPolicy: Never

and the result of kubectl describe suggestion -n kubeflow grid-example-7 is:

Name:         grid-example-7
Namespace:    kubeflow
Labels:       <none>
Annotations:  <none>
API Version:  kubeflow.org/v1beta1
Kind:         Suggestion
Metadata:
  Creation Timestamp:  2021-05-18T17:09:07Z
  Generation:          10
  Managed Fields:
    API Version:  kubeflow.org/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:ownerReferences:
      f:spec:
        .:
        f:algorithm:
          .:
          f:algorithmName:
        f:requests:
        f:resumePolicy:
      f:status:
        .:
        f:conditions:
        f:startTime:
        f:suggestionCount:
        f:suggestions:
    Manager:    katib-controller
    Operation:  Update
    Time:       2021-05-18T17:09:48Z
  Owner References:
    API Version:           kubeflow.org/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Experiment
    Name:                  grid-example-7
    UID:                   846a0212-4f8a-4e00-934e-4497297c141e
  Resource Version:        46830
  Self Link:               /apis/kubeflow.org/v1beta1/namespaces/kubeflow/suggestions/grid-example-7
  UID:                     2a6449f6-f3e8-44b9-8b9c-ea70eb2618fe
Spec:
  Algorithm:
    Algorithm Name:  grid
  Requests:          12
  Resume Policy:     LongRunning
Status:
  Conditions:
    Last Transition Time:  2021-05-18T17:09:07Z
    Last Update Time:      2021-05-18T17:09:07Z
    Message:               Suggestion is created
    Reason:                SuggestionCreated
    Status:                True
    Type:                  Created
    Last Transition Time:  2021-05-18T17:09:27Z
    Last Update Time:      2021-05-18T17:09:27Z
    Message:               Deployment is ready
    Reason:                DeploymentReady
    Status:                True
    Type:                  DeploymentReady
    Last Transition Time:  2021-05-18T17:09:48Z
    Last Update Time:      2021-05-18T17:09:48Z
    Message:               Suggestion is running
    Reason:                SuggestionRunning
    Status:                True
    Type:                  Running
  Start Time:              2021-05-18T17:09:07Z
  Suggestion Count:        9
  Suggestions:
    Name:  grid-example-7-cc64gvgz
    Parameter Assignments:
      Name:   param_1
      Value:  0
    Name:     grid-example-7-9rl4bmw8
    Parameter Assignments:
      Name:   param_1
      Value:  1
    Name:     grid-example-7-mtnrbj85
    Parameter Assignments:
      Name:   param_1
      Value:  2
    Name:     grid-example-7-ps75t8vr
    Parameter Assignments:
      Name:   param_1
      Value:  4
    Name:     grid-example-7-zhm4g28x
    Parameter Assignments:
      Name:   param_1
      Value:  5
    Name:     grid-example-7-fn5wtddt
    Parameter Assignments:
      Name:   param_1
      Value:  6
    Name:     grid-example-7-2mjwdgl6
    Parameter Assignments:
      Name:   param_1
      Value:  11
    Name:     grid-example-7-h4ml5bsp
    Parameter Assignments:
      Name:   param_1
      Value:  12
    Name:     grid-example-7-cgfnknfn
    Parameter Assignments:
      Name:   param_1
      Value:  13
Events:
  Type     Reason          Age                  From                   Message
  ----     ------          ----                 ----                   -------
  Warning  ReconcileError  14m (x22 over 102m)  suggestion-controller  The response contains unexpected trials

It seems to start okay, but at some point after the first parallel batch, the suggestion-controller hits this condition:

katib/pkg/controller.v1beta1/suggestion/suggestionclient/suggestionclient.go

Line 117 in acec1e2

if len(responseSuggestion.ParameterAssignments) != requestNum {

and the experiment gets stuck...

[sidpalas]

I did just notice another interesting thing. Even though my experiment has a single integer parameter with 15 possible values:

  parameters:
    - name: param_1
      parameterType: int
      feasibleSpace:
        min: "0"
        max: "15"

The Suggestion seems to skip some of the possible values: 0, 1, 2, 4, 5, 6, 11, 12, 13 (for this experiment it is missing 3, 7, 8, 9 ,10, 14, but the exact values + number of skipped values varies across experiments)

---

The first missing suggestion appears to always occur at parallelTrialCount + 1 (in this case param_1 = 3).

Within the suggestion pod logs I see:

INFO:pkg.suggestion.v1beta1.chocolate.base_service:New suggested parameters for Trial with chocolate_id: 3
INFO:pkg.suggestion.v1beta1.chocolate.base_service:Name = param_1, Value = 3
INFO:pkg.suggestion.v1beta1.chocolate.base_service:--------------------------------------------------

INFO:pkg.suggestion.v1beta1.chocolate.base_service:GetSuggestions returns 1 new Trials

It's almost like the suggestion is providing the correct values, but because there are already the maximum number of parallel pods, that trial never gets created. Then when another suggestion is required (because one of the initial pods completes) Chocolate continues on, skipping that value.

[sidpalas]

I copied the sqlite DB from the a suggestion pod and it shows the following:

[andreyvelich]

It's almost like the suggestion is providing the correct values, but because there are already the maximum number of parallel pods, that trial never gets created. Then when another suggestion is required (because one of the initial pods completes) Chocolate continues on, skipping that value.

Yes, you are right, because we are asking for the new Grid parameters, but Chocolate can't produce more, since budget has been reached.
Maybe we should refactor Grid suggestion to query data from the sqlite DB which has empty _trial_name before asking Chocolate to make a new sample ?

WDYT @sidpalas @gaocegege @johnugeorge ?

[johnugeorge]

I agree. This is a bug which leaks out certain trials.

[andreyvelich]

/priority p1

[gaocegege]

I agree.

[andreyvelich]

/area release