Multiple trials spawned with the same parameters when using RANDOM search

[jlewi]

/kind bug

Multiple trials appeared to be spawned using the same hyperparameters.

The search space is

  algorithm:
    algorithmName: random
    algorithmSettings: null
  ..
  parameters:
  - feasibleSpace:
      max: "5"
      min: "1"
    name: --TF_EPOCHS
    parameterType: int

I see 2 trials both of which are setting the hyperparameter TF_EPOCHS to 2.

[johnugeorge]

How many trials have you configured? MaxTrials and MaxParallelTrials

[gaocegege]

Cannot reproduce it. How many trials have you configured?

[jlewi]

I think maxTrials was like 10 and maxParallelTrials was 3. This was bigger than the search space.

[gaocegege]

I tried an experiment with 100-110 batch-size and it does not have the problem:

  Suggestions:
    Name:  tfjob-example-4vlwhsq5
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.03782690410508507
      Name:   --batch_size
      Value:  107
    Name:     tfjob-example-xgf2twtm
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.03302407575038345
      Name:   --batch_size
      Value:  108
    Name:     tfjob-example-r9rs9ffl
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.035709692926723595
      Name:   --batch_size
      Value:  107
    Name:     tfjob-example-6fbnfkl4
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.04776996787931424
      Name:   --batch_size
      Value:  102
    Name:     tfjob-example-v7bnsdrp
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.016377341645810783
      Name:   --batch_size
      Value:  100
    Name:     tfjob-example-xg4w2kv2
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.02379136549926225
      Name:   --batch_size
      Value:  102
    Name:     tfjob-example-tjjzqdp8
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.04532910237468976
      Name:   --batch_size
      Value:  101
    Name:     tfjob-example-t6b6ptr5
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.03450985419710267
      Name:   --batch_size
      Value:  107
    Name:     tfjob-example-pxn29rj8
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.03404838506896376
      Name:   --batch_size
      Value:  102
    Name:     tfjob-example-hq7cl6qj
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.04384951873857707
      Name:   --batch_size
      Value:  106
    Name:     tfjob-example-62h656kz
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.0382708329926366
      Name:   --batch_size
      Value:  105
    Name:     tfjob-example-894lcs7d
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.029473578349729884
      Name:   --batch_size
      Value:  107
    Name:     tfjob-example-b5rtrpbn
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.02560269688725199
      Name:   --batch_size
      Value:  104
    Name:     tfjob-example-4cqcz6fj
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.02169368883620422
      Name:   --batch_size
      Value:  105
    Name:     tfjob-example-qjsk6qrt
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.037560368992170676
      Name:   --batch_size
      Value:  106
    Name:     tfjob-example-m4xsf4f8
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.029494518861041187
      Name:   --batch_size
      Value:  102
    Name:     tfjob-example-h799rq28
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.0366468728744712
      Name:   --batch_size
      Value:  108
    Name:     tfjob-example-2znhvpgr
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.029767720158741892
      Name:   --batch_size
      Value:  102
    Name:     tfjob-example-5z5lncx5
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.0387312397096845
      Name:   --batch_size
      Value:  104
    Name:     tfjob-example-dpqz5zr6
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.04607817201146681
      Name:   --batch_size
      Value:  103
    Name:     tfjob-example-47q82x9g
    Parameter Assignments:
      Name:   --learning_rate
      Value:  0.0326104919821151
      Name:   --batch_size
      Value:  104

[gaocegege]

I think it depends on the random algorithm implemented by https://github.com/hyperopt/hyperopt

Now we do not maintain the implementation of random search. We reuse the implementation in hyperopt.

[johnugeorge]

I may be missing out something. In the example provided by @jlewi, TF_EPOCHS vary from 1 to 5 (integer- 5 possible values) but you had 10 trials configured? Did you have any other parameters?

[jlewi]

@johnugeorge no other parameters.

[hougangliu]

based on above, it seems it is normal behavior since after all values in the valid range applied, dup value will be applied again in new cycle.

[jlewi]

@hougangliu Could we just terminate after the cycle rather than retrying some values?

[hougangliu]

@gaocegege @johnugeorge
based on @jlewi requirement, we need enhance suggestion service to check if the new assignment has been tried (we can pass suggestion status.suggestions to algorithm service). I think the effort is not big and we can cover it soon (all algorithm service can share same function to verify new assignment) WDYT?

[johnugeorge]

@hougangliu This operation looks expensive(O(n^2)). Also does it work for types other than integer?

[hougangliu]

@hougangliu This operation looks expensive(O(n^2)). Also does it work for types other than integer?

1. I don't think n is very big to introduce performance issue.
2. also to reduce potential performance issue, we can verify it only when the length of suggestion.status.suggestions is bigger than all potential feasibleSpace range such as int, categorical and discrete type