Chocolate service db exhausted

[StefanoFioravanzo]

/kind bug

What steps did you take and what happened:
When creating experiments that use the grid search algorithm sometimes trials stop being generated, even though the experiment is still seen as running.
By printing the logs of the suggestions pods I can see the following:

ERROR:sqlalchemy.pool.impl.NullPool:Exception during reset or similar
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 680, in _finalize_fairy
    fairy._reset(pool)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 867, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 502, in do_rollback
    dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140636731225856 and this is thread id 140636053288704.
ERROR:sqlalchemy.pool.impl.NullPool:Exception closing connection <sqlite3.Connection object at 0x7fe8b4397650>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 680, in _finalize_fairy
    fairy._reset(pool)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 867, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 502, in do_rollback
    dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140636731225856 and this is thread id 140636053288704.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 270, in _close_connection
    self._dialect.do_close(connection)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in do_close
    dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140636731225856 and this is thread id 140636053288704.
DEBUG:filelock:Attempting to release lock 140636764717688 on my_db.db.lock
INFO:filelock:Lock 140636764717688 released on my_db.db.lock
INFO:BaseChocolateService:{'_chocolate_id': 0, '_loss': -83.971292, 'bm9kZXNfbnVtYmVy': 512.0}
INFO:BaseChocolateService:{'_chocolate_id': 1, '_loss': -82.655502, 'bm9kZXNfbnVtYmVy': 768.0}
DEBUG:filelock:Attempting to acquire lock 140636764717688 on my_db.db.lock
INFO:filelock:Lock 140636764717688 acquired on my_db.db.lock
DEBUG:filelock:Attempting to release lock 140636764717688 on my_db.db.lock
INFO:filelock:Lock 140636764717688 released on my_db.db.lock
INFO:BaseChocolateService:Chocolate db is exhausted, increase Search Space or decrease maxTrialCount!
INFO:BaseChocolateService:Chocolate db is exhausted, increase Search Space or decrease maxTrialCount!
INFO:BaseChocolateService:Chocolate db is exhausted, increase Search Space or decrease maxTrialCount!

Note that this is happening just sometimes, I was not able to pin down a specific configuration that causes this. With some parameters configurations it happens, with others the experiment completes by exploring the search space as expected.

What did you expect to happen:
I would at least expect to see the experiment failing (related to #1120 ), but a more informative message would be expected. The message Chocolate db is exhausted, increase Search Space or decrease maxTrialCount! doesn't make sense as the maxTrialCount is still not reaches and there are some more configurations of the search space that have not been tried.

Environment:

• Kubeflow version: 1.0
• Minikube version: 1.2.0 (MiniKF Latest)
• Kubernetes version: 1.14

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
bug	0.97

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[gaocegege]

Thanks for the issue, I will have a look

/cc @andreyvelich

[andreyvelich]

@StefanoFioravanzo Thank you for the issue.
Try to use latest image for the Grid. This PR: #1116, should fix some of these problems.
Also, can you show your Search Space and maxTrialCount?

[StefanoFioravanzo]

Writing here (after #1124 ) as it is more related:

So, now when the search space is exhausted, but the goal is not reached and also max trial count is not reaches, the suggestions pod still outputs db is exhausted, increase Search Space or decrease maxTrialCount!. It would be ideal to have this being reflected in the status of the experiment. And it should be simple to differentiate between when the maxTrialCount has been reached and when the search space was fully explored. WDYT?

I am willing to help for this and send PR, maybe can you give me some pointers?

[andreyvelich]

@StefanoFioravanzo Can you show me your yaml file with Experiment, where you got this error?

I think we should check it in Validate Algorithm Settings step.
I am not sure that using Grid user should start Experiment if maxTrialCount < count of all possible combinations of parameters.

You are right, let's discuss in this issue.

[StefanoFioravanzo]

Here it is:

apiVersion: kubeflow.org/v1alpha3
kind: Experiment
metadata:
  labels:
    controller-tools.k8s.io: '1.0'
  name: katib-test
spec:
  algorithm:
    algorithmName: grid
  parallelTrialCount: 1
  maxFailedTrialCount: 6
  maxTrialCount: 30
  objective:
    additionalMetricNames:
    goal: 100
    objectiveMetricName: result
    type: maximize
  parameters:
  - feasibleSpace:
      max: '3'
      min: '1'
    name: b
    parameterType: int
  - name: c
    parameterType: categorical
    feasibleSpace:
      list:
        - "1"
        - "9"
        - "15"
  trialTemplate:
    goTemplate:
      rawTemplate: |
        apiVersion: batch/v1
        kind: Job
        metadata:
          name: {{.Trial}}
          namespace: {{.NameSpace}}
        spec:
          backoffLimit: 0
          template:
            metadata:
              annotations:
                sidecar.istio.io/inject: "false"
            spec:
              restartPolicy: Never
              containers:
                - name: {{.Trial}}
                  image: <image>
                  command:
                    - python3 -u -c "<fn>"

This is the experiment, the search space is silly, just for testing. I omitted the image and the python command, since they are in a private repository. What I have is just a python function that returns the sum of the input arguments, that's it.

So as you can see the maxTrialCount is 30 and the result is 100, that could never be reaches. After all the trials are completed the suggestions pod raises the db exhausted error, when I would expect it to say that it has no more trials to run and exit gracefully. Maybe there could be a special experiment state for this case, when the goal could not be reached.

[andreyvelich]

@StefanoFioravanzo I believe for your search space, 30 Trials should be enough.
After using the latest image for the Grid with this Experiment you are still getting DB exhausted error?

[StefanoFioravanzo]

Exactly! I get that error when the search space is fully explored, and the goal has not been reached

[andreyvelich]

@StefanoFioravanzo I will test your search space with this amount of Trials and get back to you.

[andreyvelich]

@StefanoFioravanzo I just figure out that your Experiment has maxTrialCount more than all combinations of parameters.
I believe in your search space the maximum amount of trials is 9.

[StefanoFioravanzo]

So what you are saying is that this behaviour is expected, when the maxTrialCount is greater than all combinations?

[andreyvelich]

Yes, it is expected.
As I said above, I think we will validate it in ValidateAlgorithmSettings.