Trials with same hyperparameters - Random search #1566
Description
/kind bug
What steps did you take and what happened:
I had recently run 8 trials for an experiment and out of 8 only 4 unique combinations were seen and the rest were duplicates. The duplicates hyperparameters are seen in the majority of the experiments I run.
Ref:
#842
Looking forward to this feature. #866
What did you expect to happen:
Expected 8 different combinations of hyperparameter to run.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Suggestion pod log:
INFO:pkg.suggestion.v1beta1.hyperopt.base_service:GetSuggestions returns 8 new Trial
Suggestion controller log: (I might have missed a few as many different experiments ran after this)
{"level":"info","ts":1624915551.6826324,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
{"level":"info","ts":1624925599.2925694,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
{"level":"info","ts":1624929816.4245157,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
{"level":"info","ts":1624953809.892868,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"admin/test-123","Suggestion Requests":8,"Suggestion Count":8}
Search space:
lr: 0.001
epochs: 100
batch size: 32, 64
num of cells: 40, 60
num of layers: 2, 3
Trail configuration:
Parallel trials: 8
Max trials: 8
Max failed trails: 2
Result:
This is all I can add as of now as the pods were cleaned up. If this isn't enough please let me know what all I need to record when this happens again.
Environment:
- Kubeflow version (
kfctl version): 1.3 - Kubernetes version: (use
kubectl version): 1.19 - Katib controller version: v0.11.0