Fix MLFlow logging (#121)

This is a small change to make the MLFlow integration work. Currently
this fails with a Pandas error when trying to flatten an Omega dict; we
need to convert to a regular Python dictionary.

Can confirm this works on our MLFlow setup:
<img width="1406" height="683" alt="image"
src="https://github.com/user-attachments/assets/fcee526a-815e-4f08-bf25-d2709779ced7"
/>

Author: bthecohen

skyrl-train/docs/configuration/config.rst

@@ -121,7 +121,7 @@ Logging and Debugging Configuration
     dump_data_batch: false
     dump_eval_results: true
 
-- ``logger``: Logger to use. Currently, we support ``wandb`` and ``console``. ``console`` will simply log metrics to the console. 
+- ``logger``: Logger to use. Currently, we support ``wandb``, ``mlflow``, and ``console``. ``console`` will simply log metrics to the console. 
 - ``project_name``: Name of the project in WandB.
 - ``run_name``: Name of the run in WandB.
 - ``dump_data_batch``: Whether to dump the data batch to a file. This is useful for debugging. When ``true``, the data batch will be dumped to a file in the ``export_path`` directory. The training batch at global step ``N`` is saved to ``self.cfg.trainer.export_path / "dumped_data" / global_step_N_training_input``

skyrl-train/skyrl_train/utils/tracking.py

@@ -21,6 +21,7 @@
 from pathlib import Path
 from typing import Any, Dict, List, Union
 from loguru import logger
+from omegaconf import DictConfig, OmegaConf
 import pprint
 
 
@@ -184,6 +185,9 @@ def _compute_mlflow_params_from_objects(params) -> Dict[str, Any]:
     if params is None:
         return {}
 
+    if isinstance(params, DictConfig):
+        params = OmegaConf.to_container(params, resolve=True)
+
     return _flatten_dict(_transform_params_to_json_serializable(params, convert_list_to_dict=True), sep="/")