Skip to content

[WIP] add unit tests for katib sdk #2305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[WIP] add unit tests for katib sdk #2305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ pytest: prepare-pytest prepare-pytest-testdata
PYTHONPATH=$(PYTHONPATH) pytest ./test/unit/v1beta1/suggestion --ignore=./test/unit/v1beta1/suggestion/test_skopt_service.py
PYTHONPATH=$(PYTHONPATH) pytest ./test/unit/v1beta1/earlystopping
PYTHONPATH=$(PYTHONPATH) pytest ./test/unit/v1beta1/metricscollector
PYTHONPATH=$(PYTHONPATH) pytest ./sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py

# The skopt service doesn't work appropriately with Python 3.11.
# So, we need to run the test with Python 3.9.
Expand Down
5 changes: 3 additions & 2 deletions sdk/python/v1beta1/kubeflow/katib/api/katib_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,9 @@ def create_experiment(
f"Failed to create Katib Experiment: {namespace}/{experiment_name}"
)

# TODO (andreyvelich): Use proper logger.
print(f"Experiment {namespace}/{experiment_name} has been created")
success_message = f"Experiment {namespace}/{experiment_name} has been created"
print(success_message)
return success_message
Comment on lines +132 to +134
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need this change ?


if self._is_ipython():
if self.in_cluster:
Expand Down
55 changes: 55 additions & 0 deletions sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import multiprocessing
import pytest
from unittest.mock import patch, Mock
from kubeflow.katib import models
from kubeflow.katib.api.katib_client import KatibClient

@pytest.fixture
def katib_client():
return KatibClient()

@pytest.fixture
def valid_experiment():
return models.V1beta1Experiment(
metadata=models.V1ObjectMeta(name="test-experiment"),
spec=models.V1beta1ExperimentSpec()
)

class TestCreateExperiment:
@patch('kubeflow.katib.api.katib_client.client.CustomObjectsApi.create_namespaced_custom_object')
def test_create_experiment_success(self, mock_create_namespaced_custom_object, katib_client, valid_experiment):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, it would be nice if you could add unit test to verify these conditions: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L96-L108.
E.g. you can create create test_data list under TestCreateExperiment class and add various test cases there, similar to TrainingClient tests.

mock_create_namespaced_custom_object.return_value = {"metadata": {"name": "test-experiment"}}
experiment = katib_client.create_experiment(valid_experiment)
assert experiment == "Experiment default/test-experiment has been created"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create_experiment API doesn't return any info, so you should not check this assert.


@patch('kubeflow.katib.api.katib_client.client.CustomObjectsApi.create_namespaced_custom_object')
def test_create_experiment_timeout_error(self, mock_create_namespaced_custom_object, katib_client, valid_experiment):
mock_create_namespaced_custom_object.side_effect = multiprocessing.TimeoutError()
with pytest.raises(TimeoutError):
katib_client.create_experiment(valid_experiment, namespace="timeout")

@patch('kubeflow.katib.api.katib_client.client.CustomObjectsApi.create_namespaced_custom_object')
def test_create_experiment_runtime_error(self, mock_create_namespaced_custom_object, katib_client, valid_experiment):
mock_create_namespaced_custom_object.side_effect = RuntimeError()
with pytest.raises(RuntimeError):
katib_client.create_experiment(valid_experiment, namespace="runtime")
Comment on lines +28 to +35
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we testing here that create_experiment raises correct exception based on create_namespaced_custom_object' call ?


class TestGetExperiment:
@patch('kubeflow.katib.api.katib_client.client.CustomObjectsApi.get_namespaced_custom_object')
def test_get_experiment_success(self, mock_get_namespaced_custom_object, katib_client, valid_experiment):
mock_response = {"metadata": {"name": "test-experiment"}, "spec": {}, "status": {}}
mock_get_namespaced_custom_object.return_value = mock_response
experiment = katib_client.get_experiment("test-experiment")
assert experiment == valid_experiment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_get_experiment_success is failing currently, can you help me fix this?

================================================================ short test summary info ================================================================
FAILED sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py::TestGetExperiment::test_get_experiment_success - AssertionError: assert None == {'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': None,\n              'cluster_name': None,\n     ...
======================================================= 1 failed, 5 passed, 100 warnings in 0.28s =======================================================
make: *** [pytest] Error 1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I can see, get_experiment returns null object, did you mock the call correctly ?
Should you add patches on the KatibClient, similar as in TrainingClient: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client_test.py#L181-L195 ?

@patch('kubeflow.katib.api.katib_client.client.CustomObjectsApi.get_namespaced_custom_object')
def test_get_experiment_timeout_error(self, mock_get_namespaced_custom_object, katib_client):
mock_get_namespaced_custom_object.side_effect = multiprocessing.TimeoutError()
with pytest.raises(TimeoutError):
katib_client.get_experiment("test-experiment", namespace="timeout")

@patch('kubeflow.katib.api.katib_client.client.CustomObjectsApi.get_namespaced_custom_object')
def test_get_experiment_runtime_error(self, mock_get_namespaced_custom_object, katib_client):
mock_get_namespaced_custom_object.side_effect = RuntimeError()
with pytest.raises(RuntimeError):
katib_client.get_experiment("test-experiment", namespace="runtime")