Katib Roadmap 2021

[andreyvelich]

We should discuss our Katib Roadmap for 2021.

Some points from my side:

Katib enhancements

• Finish AWS CI/CD migration for the release process.
• New Katib UI implementation: https://github.com/kubeflow/katib/projects/1.
• Support multiple versions in Katib CRDs.
• Support various parameter distribution: [feature] Support log-uniform scale in search space definition #1207.
• Finish validation for the algorithms: Add validation for algortihms (a.k.a suggestions) #1126.

Advanced AutoML features

• Refactor Hyperband: Refactor Hyperband Suggestion #1389.
• Support ASHA, I believe it is very popular.
• Separate CRD for NAS and HP ?
• Support Population Based Training: Support for PBT (Population Based Training) #1382.
• Support AutoMC: [feature request] Support AutoMC in Katib #460.
• Support Feature Engineering: [feature request] Support automated feature engineering in Katib #475.

Please give your feedback and what do you want to see in Katib 2021.
cc @gaocegege @johnugeorge @PatrickXYS @terrytangyuan @kimwnasptd @yanniszark @jbottum @c-bata @RFMVasconcelos @knkski @anneum @thomas-mueller

Let's finalise our Roadmap on the upcoming WG meetings (24th of March and 7th of April).

[PatrickXYS]

This is awesome! Getting a roadmap and get all the items shipped at the end of the 2021 will be a great milestone

[c-bata]

I think it is great if Katib dashboard will support a visualization of hyperparameter importance using fANOVA algorithm.

• https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.plot_param_importances.html#optuna.visualization.plot_param_importances
• https://www.automl.org/algorithm-analysis/fanova/

optuna-dashboard will support it from the next release:
optuna/optuna-dashboard#54 (comment)

And I'm planning to implement fANOVA algorithm in Goptuna next or the following month 👍

[andreyvelich]

It's great idea @c-bata.
I think we can try to include this feature in the new Katib UI.
cc @kimwnasptd

Btw, @c-bata do we have plans to integrate #1283 in Katib 2021 ?

[c-bata]

do we have plans to integrate #1283 in Katib 2021 ?

Ah, yes! Sorry, I forgot it.

[andreyvelich]

Support MLMD for Katib Experiment.

[frankmanbb]

TFX also has a tuner, https://www.tensorflow.org/tfx/guide/tuner, any possibility that we can use Katib in tfx as well? Maybe as a custom tfx component?

[andreyvelich]

TFX also has a tuner, https://www.tensorflow.org/tfx/guide/tuner, any possibility that we can use Katib in tfx as well? Maybe as a custom tfx component?

This is interesting. That should be possible when TFX is running on top of Kubernetes.
WDYT @gaocegege @johnugeorge ?
@Bobgy Do we have any examples when KFP and TFX are running together ?