[GSOC] Project 8: Support various Parameter Distribution in Katib #2374
Description
Motivation
Currently, Katib is limited to supporting only uniform distribution for integer, float, and categorical hyperparameters. By introducing additional distributions, Katib will become more flexible and powerful in conducting hyperparameter optimization tasks.
A Data Scientist requires Katib to support multiple hyperparameter distributions, such as log-uniform, normal, and log-normal, in addition to the existing uniform distribution. For instance, learning rates often benefit from a log-uniform distribution because small values can significantly impact performance. Similarly, normal distributions are useful for parameters that are expected to vary around a central value.
Goal
The project aims to enhance Katib's Experiment APIs by introducing support for various hyperparameter distributions beyond the current uniform distribution.
Currently, the Experiment APIs support parameterType and feasibleSpace. To provide greater flexibility, we expanded the feasibleSpace to include a Distribution field, allowing users to specify the type of distribution for each parameter.
This Distribution field is designed to be mutually exclusive, meaning it can be left empty, enabling users to tune hyperparameters without specifying a distribution if desired.
What I Did in GSoC Project & Ongoing Works
Preparation
- Converted the GSoC proposal into a KEP: [GSOC] Support for various Parameter distributions in Katib #2334
Development
- Added
Distributionfield to theFeasibleSpaceobject: [GSoC] AddedDistributionTypeto Experiment API #2377 - Updated gRPC APIs to include the
Distributionfield: addedDistributionfield to feasibleSpace inapi.proto#2397 - Modified Suggestion client to handle the new Distribution field: [GSOC] added Unknown distribution and convertDistribution in suggestion client #2403
- Updated validation webhook to support both ParameterType and Distribution: [GSOC] Add validator for feasible space distribution #2404
- Implemented logic updates for handling new distributions in
hyperoptsuggestion service: [GSOC]hyperoptsuggestion service logic update #2412 - Implemented logic updates for handling new distributions in
optunasuggestion service - Implemented logic updates for handling new distributions in
goptunasuggestion service
Documentation and Ongoing Work
- Write e2e tests for
normalandlognormaldistributions inhyperopt: [GSOC]hyperoptsuggestion service logic update #2412 - Extend the Python SDK to include support for the
Distributionfield. - Update Katib documentation to reflect new API changes and usage guidelines.
- Create examples demonstrating the use of different parameter distributions.
What I learnt during the project
-
One of the key hurdles was integrating multiple new distributions into the Katib Experiment APIs while ensuring backward compatibility. Modifying the
feasibleSpaceto include aDistributionfield required careful consideration of API design and how different suggestion services, like Optuna and Hyperopt, would handle these changes. -
Effective communication is key. The weekly syncs with @andreyvelich and @tenzen-y were extremely valuable, and having a clear agenda beforehand helped me get my questions answered efficiently and stay on track.
-
I learned the importance of end-to-end (E2E) tests. By implementing these tests, I gained valuable insights into how the different components of Katib interact and how critical it is to verify that new features, like the distribution support.
In the End
I am grateful to Google for the opportunity to work on this project and to the Kubeflow community for their support and guidance. Special thanks to my project mentors, @andreyvelich, @tenzen-y, and @terrytangyuan for your invaluable insights and encouragement. Your mentorship was pivotal in achieving the project's goals.
I hope the enhancements made will significantly benefit users and advance Katib’s capabilities in hyperparameter optimization. Looking forward to continue to contribute to Kubeflow.