Update README and out-of-date docs (#2438)

Signed-off-by: Andrey Velichkevich <[email protected]>

Author: andreyvelich

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,6 +1,6 @@
 <!--  Thanks for sending a pull request! Here are some tips for you:
 1. If this is your first time, check our contributor guidelines https://www.kubeflow.org/docs/about/contributing
-2. To know more about Katib components, check developer guide https://github.com/kubeflow/katib/blob/master/docs/developer-guide.md
+2. To know more about Katib components, check developer guide https://github.com/kubeflow/katib/blob/master/CONTRIBUTING.md
 3. If you want *faster* PR reviews, check how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
 -->
 

docs/developer-guide.md renamed to CONTRIBUTING.md

@@ -2,13 +2,12 @@
 
 This developer guide is for people who want to contribute to the Katib project.
 If you're interesting in using Katib in your machine learning project,
-see the following user guides:
+see the following guides:
 
-- [Concepts](https://www.kubeflow.org/docs/components/katib/overview/)
-  in Katib, hyperparameter tuning, and neural architecture search.
 - [Getting started with Katib](https://kubeflow.org/docs/components/katib/hyperparameter/).
-- Detailed guide to [configuring and running a Katib
-  experiment](https://kubeflow.org/docs/components/katib/experiment/).
+- [How to configure Katib Experiment](https://kubeflow.org/docs/components/katib/experiment/).
+- [Katib architecture and concepts](https://www.kubeflow.org/docs/components/katib/reference/architecture/)
+  for hyperparameter tuning and neural architecture search.
 
 ## Requirements
 
@@ -93,10 +92,6 @@ Below is a list of command-line flags accepted by Katib DB Manager:
 | --------------- | ------------- | ------- | ------------------------------------------------------- |
 | connect-timeout | time.Duration | 60s     | Timeout before calling error during database connection |
 
-## Workflow design
-
-Please see [workflow-design.md](./workflow-design.md).
-
 ## Katib admission webhooks
 
 Katib uses three [Kubernetes admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/).
@@ -113,7 +108,7 @@ Katib uses three [Kubernetes admission webhooks](https://kubernetes.io/docs/refe
 1. `mutator.pod.katib.kubeflow.org` - Mutating admission webhook to inject the metrics
    collector sidecar container to the training pod. Learn more about the Katib's
    metrics collector in the
-   [Kubeflow documentation](https://www.kubeflow.org/docs/components/katib/experiment/#metrics-collector).
+   [Kubeflow documentation](https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/).
 
 You can find the YAMLs for the Katib webhooks
 [here](../manifests/v1beta1/components/webhook/webhooks.yaml).
@@ -168,4 +163,4 @@ they'll be executed against every file in the repository.
 
 Specific programmatically generated files listed in the `exclude` field in
 [.pre-commit-config.yaml](../.pre-commit-config.yaml) are deliberately excluded
-from the hooks. 
+from the hooks.

README.md

@@ -29,13 +29,13 @@ and many more.
 
 Katib stands for `secretary` in Arabic.
 
-# Search Algorithms
+## Search Algorithms
 
 Katib supports several search algorithms. Follow the
-[Kubeflow documentation](https://www.kubeflow.org/docs/components/katib/experiment/#search-algorithms-in-detail)
+[Kubeflow documentation](https://www.kubeflow.org/docs/components/katib/user-guides/hp-tuning/configure-algorithm/#hp-tuning-algorithms)
 to know more about each algorithm and check the
-[Suggestion service guide](/docs/new-algorithm-service.md) to implement your
-custom algorithm.
+[this guide](https://www.kubeflow.org/docs/components/katib/user-guides/hp-tuning/configure-algorithm/#use-custom-algorithm-in-katib)
+to implement your custom algorithm.
 
 <table>
   <tbody>
@@ -137,141 +137,68 @@ custom algorithm.
   </tbody>
 </table>
 
-To perform above algorithms Katib supports the following frameworks:
+To perform the above algorithms Katib supports the following frameworks:
 
 - [Goptuna](https://github.com/c-bata/goptuna)
 - [Hyperopt](https://github.com/hyperopt/hyperopt)
 - [Optuna](https://github.com/optuna/optuna)
 - [Scikit Optimize](https://github.com/scikit-optimize/scikit-optimize)
 
-# Installation
-
-For the various Katib installs check the
-[Kubeflow guide](https://www.kubeflow.org/docs/components/katib/hyperparameter/#katib-setup).
-Follow the next steps to install Katib standalone.
-
 ## Prerequisites
 
-This is the minimal requirements to install Katib:
-
-- Kubernetes >= 1.27
-- `kubectl` >= 1.27
+Please check [the official Kubeflow documentation](https://www.kubeflow.org/docs/components/katib/installation/#prerequisites)
+for prerequisites to install Katib.
 
-## Latest Version
+## Installation
 
-For the latest Katib version run this command:
-
-```
-kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=master"
-```
+Please follow [the Kubeflow Katib guide](https://www.kubeflow.org/docs/components/katib/installation/#installing-katib)
+for the detailed instructions on how to install Katib.
 
-## Release Version
+### Installing the Control Plane
 
-For the specific Katib release (for example `v0.14.0`) run this command:
+Run the following command to install the latest stable release of Katib control plane:
 
 ```
-kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=v0.14.0"
+kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=v0.17.0"
 ```
 
-Make sure that all Katib components are running:
+Run the following command to install the latest changes of Katib control plane:
 
 ```
-$ kubectl get pods -n kubeflow
-
-NAME                                READY   STATUS      RESTARTS   AGE
-katib-controller-566595bdd8-hbxgf   1/1     Running     0          36s
-katib-db-manager-57cd769cdb-4g99m   1/1     Running     0          36s
-katib-mysql-7894994f88-5d4s5        1/1     Running     0          36s
-katib-ui-5767cfccdc-pwg2x           1/1     Running     0          36s
+kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=master"
 ```
 
 For the Katib Experiments check the [complete examples list](./examples/v1beta1).
 
-# Quickstart
+### Installing the Python SDK
 
-You can run your first HyperParameter Tuning Experiment using [Katib Python SDK](./sdk/python/v1beta1).
+Katib implements [a Python SDK](https://pypi.org/project/kubeflow-katib/) to simplify creation of
+hyperparameter tuning jobs for Data Scientists.
 
-In the following example we are going to maximize a simple objective function:
-$F(a,b) = 4a - b^2$. The bigger $a$ and the lesser $b$ value, the bigger the function value $F$.
+Run the following command to install the latest stable release of Katib SDK:
 
-```python
-import kubeflow.katib as katib
-
-# Step 1. Create an objective function.
-def objective(parameters):
-    # Import required packages.
-    import time
-    time.sleep(5)
-    # Calculate objective function.
-    result = 4 * int(parameters["a"]) - float(parameters["b"]) ** 2
-    # Katib parses metrics in this format: <metric-name>=<metric-value>.
-    print(f"result={result}")
-
-# Step 2. Create HyperParameter search space.
-parameters = {
-    "a": katib.search.int(min=10, max=20),
-    "b": katib.search.double(min=0.1, max=0.2)
-}
-
-# Step 3. Create Katib Experiment.
-katib_client = katib.KatibClient()
-name = "tune-experiment"
-katib_client.tune(
-    name=name,
-    objective=objective,
-    parameters=parameters,
-    objective_metric_name="result",
-    max_trial_count=12
-)
-
-# Step 4. Get the best HyperParameters.
-print(katib_client.get_optimal_hyperparameters(name))
+```sh
+pip install -U kubeflow-katib
 ```
 
-# Documentation
-
-- Check
-  [the Katib getting started guide](https://www.kubeflow.org/docs/components/katib/hyperparameter/#example-using-random-search-algorithm).
-
-- Learn about Katib **Concepts** in this
-  [guide](https://www.kubeflow.org/docs/components/katib/overview/#katib-concepts).
-
-- Learn about Katib **Interfaces** in this
-  [guide](https://www.kubeflow.org/docs/components/katib/overview/#katib-interfaces).
-
-- Learn about Katib **Components** in this
-  [guide](https://www.kubeflow.org/docs/components/katib/hyperparameter/#katib-components).
-
-- Know more about Katib in the [presentations and demos list](./docs/presentations.md).
+## Getting Started
 
-# Community
+Please refer to [the getting started guide](https://www.kubeflow.org/docs/components/katib/getting-started/#getting-started-with-katib-python-sdk)
+to quickly create your first hyperparameter tuning Experiment using the Python SDK.
 
-We are always growing our community and invite new users and AutoML enthusiasts
-to contribute to the Katib project. The following links provide information
-about getting involved in the community:
+## Community
 
-- Subscribe to the
-  [AutoML calendar](https://calendar.google.com/calendar/u/0/r?cid=ZDQ5bnNpZWZzbmZna2Y5MW8wdThoMmpoazRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ)
-  to attend Working Group bi-weekly community meetings.
+The following links provide information on how to get involved in the community:
 
-- Check the
-  [AutoML and Training Working Group meeting notes](https://docs.google.com/document/d/1MChKfzrKAeFRtYqypFbMXL6ZIc_OgijjkvbqmwRV-64/edit).
-
-- If you use Katib, please update [the adopters list](ADOPTERS.md).
+- Attend [the bi-weekly AutoML and Training Working Group](https://bit.ly/2PWVCkV)
+  community meeting.
+- Join our [`#kubeflow-katib`](https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels)
+  Slack channel.
+- Check out [who is using Katib](ADOPTERS.md) and [presentations about Katib project](docs/presentations.md).
 
 ## Contributing
 
-Please feel free to test the system! [Developer guide](./docs/developer-guide.md)
-is a good starting point for our developers.
-
-## Blog posts
-
-- [Kubeflow Katib: Scalable, Portable and Cloud Native System for AutoML](https://blog.kubeflow.org/katib/)
-  (by Andrey Velichkevich)
-
-## Events
-
-- [AutoML and Training WG Summit. 16th of July 2021](https://docs.google.com/document/d/1vGluSPHmAqEr8k9Dmm82RcQ-MVnqbYYSfnjMGB-aPuo/edit?usp=sharing)
+Please refer to the [CONTRIBUTING guide](CONTRIBUTING.md).
 
 ## Citation
 

docs/README.md

@@ -0,0 +1,5 @@
+# Katib Documentation
+
+Welcome to Kubeflow Katib!
+
+The Katib documentation is available on [kubeflow.org](https://www.kubeflow.org/docs/components/katib/).

docs/images-location.md

@@ -5,7 +5,7 @@ Here you can find the location for images that are used in Katib.
 ## Katib Components Images
 
 The following table shows images for the
-[Katib components](https://www.kubeflow.org/docs/components/katib/hyperparameter/#katib-components).
+[Katib components](https://www.kubeflow.org/docs/components/katib/reference/architecture/#katib-control-plane-components).
 
 <table>
   <tbody>
@@ -70,7 +70,7 @@ The following table shows images for the
 ## Katib Metrics Collectors Images
 
 The following table shows images for the
-[Katib Metrics Collectors](https://www.kubeflow.org/docs/components/katib/experiment/#metrics-collector).
+[Katib Metrics Collectors](https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/).
 
 <table>
   <tbody>
@@ -113,8 +113,8 @@ The following table shows images for the
 ## Katib Suggestions and Early Stopping Images
 
 The following table shows images for the
-[Katib Suggestions](https://www.kubeflow.org/docs/components/katib/experiment/#search-algorithms-in-detail)
-and the [Katib Early Stopping algorithms](https://www.kubeflow.org/docs/components/katib/early-stopping/).
+[Katib Suggestion services](https://www.kubeflow.org/docs/components/katib/reference/architecture/#suggestion)
+and the [Katib Early Stopping algorithms](https://www.kubeflow.org/docs/components/katib/user-guides/early-stopping/#early-stopping-algorithms).
 
 <table>
   <tbody>
@@ -223,7 +223,7 @@ and the [Katib Early Stopping algorithms](https://www.kubeflow.org/docs/componen
 ## Training Containers Images
 
 The following table shows images for training containers which are used in the
-[Katib Trials](https://www.kubeflow.org/docs/components/katib/experiment/#packaging-your-training-code-in-a-container-image).
+[Katib Trials](https://www.kubeflow.org/docs/components/katib/reference/architecture/#trial).
 
 <table>
   <tbody>