Skip to content

Conversation

lijm1358
Copy link
Contributor

@lijm1358 lijm1358 commented Jan 7, 2024

Fix the issue of Optuna tuning mode with LightGBM, which always fails after LightGBM 4.0.0. verbose_eval has been removed as it did nothing if False was passed; early_stopping_rounds has also been removed and made into a callback.
All tests passed, and also tested with lightgbm==3.3.5, as using the older version of lightgbm was one of the solution in #645.

  • before changes:
AutoML directory: test_output
Expected computing time:
Time for tuning with Optuna: len(algorithms) * optuna_time_budget = 21600 seconds
There is no time limit for ML model training after Optuna tuning (total_time_limit parameter is ignored).        
The task is multiclass_classification with evaluation metric logloss
AutoML will use algorithms: ['Random Forest', 'Extra Trees', 'LightGBM', 'Xgboost', 'CatBoost', 'Neural Network']
AutoML will stack models
AutoML will ensemble available models
AutoML steps: ['simple_algorithms', 'default_algorithms', 'ensemble', 'stack', 'ensemble_stacked']
Skip simple_algorithms because no parameters were generated.
* Step default_algorithms will try to check up to 6 models
Optuna optimizes LightGBM with time budget 3600 seconds eval_metric logloss (minimize)
[I 2024-01-08 00:30:52,426] A new study created in memory with name: no-name-e1b40c1d-77c0-4cff-bf8e-78f1677c5906
Exception in LightgbmObjective train() got an unexpected keyword argument 'verbose_eval'
[W 2024-01-08 00:30:52,477] Trial 0 failed with parameters: {'learning_rate': 0.1, 'num_leaves': 1598, 'lambda_l1': 2.840098794801191e-06, 'lambda_l2': 3.0773599420974e-06, 'feature_fraction': 0.8613105322932351, 'bagging_fraction': 0.970697557159987, 'bagging_freq': 7, 'min_data_in_leaf': 36, 'extra_trees': False} because of the following error: The value None could not 
be cast to float..
[W 2024-01-08 00:30:52,478] Trial 0 failed with value None.
Exception in LightgbmObjective train() got an unexpected keyword argument 'verbose_eval'
[W 2024-01-08 00:30:52,581] Trial 1 failed with parameters: {'learning_rate': 0.0125, 'num_leaves': 30, 'lambda_l1': 0.09024841733204539, 'lambda_l2': 0.8785585624049705, 'feature_fraction': 0.5554201923798203, 'bagging_fraction': 0.7307773310574073, 'bagging_freq': 1, 'min_data_in_leaf': 37, 'extra_trees': True} because of the following error: The value None could not be cast to float..
[W 2024-01-08 00:30:52,582] Trial 1 failed with value None.
...
  • after changes
[I 2024-01-08 00:32:03,378] A new study created in memory with name: no-name-516d0a9b-d37c-4a2d-811b-856f69d57464
[I 2024-01-08 00:32:03,556] Trial 0 finished with value: 0.1464389479127149 and parameters: {'learning_rate': 0.1, 'num_leaves': 1598, 'lambda_l1': 2.840098794801191e-06, 'lambda_l2': 3.0773599420974e-06, 'feature_fraction': 0.8613105322932351, 'bagging_fraction': 0.970697557159987, 'bagging_freq': 7, 'min_data_in_leaf': 36, 'extra_trees': False}. Best is trial 0 with value: 0.1464389479127149.
[I 2024-01-08 00:32:04,168] Trial 1 finished with value: 0.356323461032784 and parameters: {'learning_rate': 0.0125, 'num_leaves': 30, 'lambda_l1': 0.09024841733204539, 'lambda_l2': 0.8785585624049705, 'feature_fraction': 0.5554201923798203, 'bagging_fraction': 0.7307773310574073, 'bagging_freq': 1, 'min_data_in_leaf': 37, 'extra_trees': True}. Best is trial 0 with value: 0.1464389479127149.
[I 2024-01-08 00:32:05,511] Trial 2 finished with value: 0.11546304145125812 and parameters: {'learning_rate': 0.025, 'num_leaves': 1781, 'lambda_l1': 8.42482357544477e-05, 'lambda_l2': 0.1657023923779856, 'feature_fraction': 0.4006367785978634, 'bagging_fraction': 0.7929826868254444, 'bagging_freq': 5, 'min_data_in_leaf': 22, 'extra_trees': True}. Best is trial 2 with value: 0.11546304145125812.
[I 2024-01-08 00:32:05,804] Trial 3 finished with value: 0.9447620251350354 and parameters: {'learning_rate': 0.0125, 'num_leaves': 1383, 'lambda_l1': 0.0022471032062799873, 'lambda_l2': 
0.0006306544535695586, 'feature_fraction': 0.3303268443196031, 'bagging_fraction': 0.6930031616587092, 'bagging_freq': 3, 'min_data_in_leaf': 51, 'extra_trees': False}. Best is trial 2 with value: 0.11546304145125812.
...

related to: #683, #645

@pplonski pplonski merged commit 17af68b into mljar:master Jan 8, 2024
@pplonski
Copy link
Contributor

pplonski commented Jan 8, 2024

Thank you @lijm1358!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants