DummyClassifier wins for no obvious reasons

[adamhaber-atidot]

Hello,

I'm experiencing difficulties similar to #140 - I'm trying to fit an autosklearn classifier, for which I got reasonable results using sgd logistic regression/xgboost, but all I get is an ensemble containing the DummyClassifier.

Some relevant details:

• Autosklearn version - 0.4.0 - from the release notes I was under the impression that Should baseline DummyClassifier ever be the winning ensemble? #140 should be resolved...
• Tried subsampling.
• Tried increasing the time limit (per run and per task) to a couple of hours - sgd logisitic regression takes less than a minute.
• Tried dense/sparse representations.
• The problem is solved by using AUC loss (as in Should baseline DummyClassifier ever be the winning ensemble? #140), but this isn't very helpful - I'm trying to optimize a custom loss function (for which the problem persists).
• Log loss also isn't working.

Any ideas? Maybe it's possible to exclude the DummyClassifier?

[mfeurer]

A solution for which only the DummyClassifier is considered good suggests that evaluating the machine learning algorithm fails. Please check the log files for error messages. If possible, could you also provide a minimal example on an example dataset using your loss function?

[adamhaber-atidot]

I first suspected that the error is in my custom loss function, but after the problem reproduced with the log loss, I reckoned the problem is elsewhere...

I've inspected the log file and haven't found any error messages. Strangely, I don't find "finished evaluating" messages - neither in the AUC case (which fits non-dummy estimators) and the logloss/custom_func case (which fits only the dummy estimator). Is there anything specific in the log file which would be informative to put here?

[mfeurer]

Could you please share the logfile on something like pastebin or upload it to github?

[adamhaber-atidot]

Sure - these two are for very short runs, just to reproduce the problem (but the same problem happens when the time limit is hours).

This is for an AUC metric run (converged):
https://pastebin.com/3fSVjDgy

This is for a log_loss run (fits dummy):
https://pastebin.com/e8ReSwpL

[mfeurer]

This is indeed a bug with loss functions (there seems to be no issue with score functions). If you remove the line sorted_keys = filter(lambda x: x[1] > 0.001, sorted_keys) it should work for now.

[adamhaber-atidot]

I rewrote the function to be a score and not a loss and now it works fine - Thanks!

[kaiweiang]

I rewrote the function to be a score and not a loss and now it works fine - Thanks!

@adamhaber-atidot Do you mind sharing the custom function you wrote that makes log_loss work? Thanks

[mfeurer]

Will be fixed in the next release - for now you can use the development branch.