Support for r2 metric in Optuna mode

[Possums]

Currently, r2 metric evaluation is not supported in the tuner/optuna/tuner.py file.

if eval_metric.name not in ["auc", "logloss", "rmse", "mae", "mape"]: raise AutoMLException(f"Metric {eval_metric.name} is not supported")

When I manually add 'r2' to the list, I encounter the following error.

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/supervised/base_automl.py", line 1054, in _fit trained = self.train_model(params) File "/usr/local/lib/python3.8/dist-packages/supervised/base_automl.py", line 356, in train_model mf.train(results_path, model_subpath) File "/usr/local/lib/python3.8/dist-packages/supervised/model_framework.py", line 185, in train self.learner_params = optuna_tuner.optimize( File "/usr/local/lib/python3.8/dist-packages/supervised/tuner/optuna/tuner.py", line 106, in optimize objective = LightgbmObjective( File "/usr/local/lib/python3.8/dist-packages/supervised/tuner/optuna/lightgbm.py", line 61, in __init__ self.eval_metric_name = metric_name_mapping[ml_task][self.eval_metric.name] KeyError: 'r2'

Is this a known limitation, and if so, is there a way to work around it?

[pplonski]

Currently only ["auc", "logloss", "rmse", "mae", "mape"] metrics are supported by Optuna mode. The R2 is not supported because there were no native support for R2 metric in Xgboost, CatBoost, and LightGBM.

This can be fixed by adding a custom eval_metric.

@Possums do you have other metrics in mind worth adding?

[Possums]

Thanks for the quick response!

I would say other metrics that I would find useful are spearman correlation and RMSLE for regression, and perhaps AUCPR and F1 for classification.

[pplonski]

@Possums sounds good, let me check the details which metrics can be added soon

[pplonski]

I've added support to r2 metric to Optuna mode. The code is pushed to the dev branch.

To try it please install mljar-supervised from the dev branch:

pip install -U git+https://github.com/mljar/mljar-supervised.git@dev

I will add the rest of the metrics as well (spearman, rmsle, aucpr, f1). For f1 metric there can be two implementations:

• with static 0.5 threshold,
• with automatic threshold adjustment.
  For now I will implement the 0.5 threshold version - it is simpler.

[Possums]

Thanks for the quick update! I'm currently training a model and haven't run into any errors so far.

[pplonski]

I've added:

• r2, spearman, and pearson correlation metrics for regression tasks,
• f1 score for binary and multi-class classification (Micro average),
• average_precision for binary classification (AUCPR).

I didn't implement RMSLE - need to update target preprocessing for this to assure positive values only. I will add a ticket for this metric (#346).

All changes are in the dev branch. To install package directly from the dev branch:

pip install -U git+https://github.com/mljar/mljar-supervised.git@dev