`AutoML::fit_ensemble` with `ensemble_size =0` causes crash

[eddiebergman]

It seems there is no validation on fit_ensemble when ensemble size is 0, causing an issue to appear as seen in #1327

[bkpcoding]

Hey, I am looking to contribute to this issue. Currently, I am going through the contributing guidelines and setting up the development environment.
So as far as I understand the current show_models() function crashes if the ensemble_size = 0 and this needs to be addressed by just returning an empty dictionary? Or there has to be an error or warning message?

[eddiebergman]

Hi @bkpcoding,

Turns out this is wrong, it has nothing to do with show_models() and I think it has actually to do with fit_ensemble(train_y, ensemble_size=0) in the provided issue. The extra code in the issue was misleading to me and I will update the post and title.

I still think this issue might be easy enough though, having a quick look at autosklearn/automl/AutoML::fit_ensemble and tracing through a little bit, it seems there's no real check for the ensemble_size parameter.

If you'd like to contribute anything there, please do :)

[bkpcoding]

What exact behaviour are we expecting if ensemble_size = 0? Are we supposed to show all the models trained when the show_models() function is called?
I think if the ensemble size is 0 and the user calls fit_ensemble we can raise a warning stating that given ensemble size is zero and return None and in the show_models function we can return an empty dictionary or raise an error that you show_models only returns ensemble models or we can return all the trained models when show_models is called.
However, I was not able to reproduce this error from #1327

automl = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=30,
    ensemble_size=0
)
automl.fit(X_train, y_train, dataset_name='breast_cancer')
automl.sprint_statistics()
automl.show_models()

[WARNING] [2021-12-27 12:14:01,525:Client-autosklearn.util.backend] Directory /tmp/autosklearn_tmp_0d28300d-66e0-11ec-bfa6-b365af5c825c/.auto-sklearn/ensembles does not exist
Dataset name: 0d28293e-66e0-11ec-bfa6-b365af5c825c\n  Metric: accuracy\n  Best validation score: 0.966667\n  Number of target algorithm runs: 67\n  Number of successful target algorithm runs: 67\n  Number of crashed target algorithm runs: 0
...
...
line 1542, in _load_models
    raise ValueError('No models fitted!')
ValueError: No models fitted!

Instead I got this one,

Traceback (most recent call last):
  File "autosk_ex.py", line 27, in <module>
    print(automl.show_models())
  File "/home/sagar/auto-sklearn/autosklearn/estimators.py", line 608, in show_models
    return self.automl_.show_models()
  File "/home/sagar/auto-sklearn/autosklearn/automl.py", line 1925, in show_models
    for i, weight in enumerate(self.ensemble_.weights_):
AttributeError: 'NoneType' object has no attribute 'weights_'

[eddiebergman]

Hey, thanks for trying to reproduce the error, the unreproducibilty of it means it's probably related to now models being found for the dataset in that time limit.

I would honestly advocate for the simplest solutions:

• fit_ensemble gives a ValueError with ensemble_size = 0.

• As for show_models(), I'm currently on my phone and can't easily check, but I recall it only shows Ensemble members. It would be good to extend this to all models but I think for simplicity sake, a warning and an empty dict will suffice.

I have been working on a generic backend for auto-sklearn and auto-pytorch at automl_common which should help with model and ensemble interaction. This is in the pipeline for the end of this month so I wouldn't spend to much time on perfecting this PR.