Python SDK for Kubeflow Training Operator

[alembiewski]

Are there any plans to add Python SDK for the new all-in-one Kubeflow Training Operator?

[alembiewski]

/kind question

[gaocegege]

/cc @kubeflow/wg-training-leads

[gaocegege]

I think we should have it, but do not have the bandwidth for it now.

[johnugeorge]

Yes.
/help

[google-oss-robot]

@johnugeorge:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Yes.
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Jeffwan]

FYI

Ketan ping me earlier and seems Flyte user wants to use it to submit jobs. flyteorg/flyte#1375

[Jeffwan]

@alembiewski Do you have any specific requirements? Existing SDK should work out of the box since we have not changed API yet. If that works for you, we can consider to extend it to other frameworks. If not, what level of abstraction do you need?

[alembiewski]

@Jeffwan, I think by having the unified SDK that supports multiple frameworks we could possibly reduce code duplication between client APIs by introducing a generic API client, which supports multiple model types (pytorchjob api and tfjob api look really similar to each other) - this approach will streamline the process of adding new frameworks to the SDK, but could be tricky to implement.

The current model with separate SDK per framework works for us as well, although there are few serious limitations:

• kubernetes client version lock-in makes the SDK incompatible with the other Kubeflow SDKs, such as KFserving 0.5.1, KFP with the upcoming Katib SDK.
• inconsistencies between the API and SDK (e.g.: Update swagger.json schema for TFJobSpec to include RunPolicy #1278)

It seems to me that such limitations could be dropped just by regenerating the SDK models with OpenAPI generator as it was done recently for Katib: kubeflow/katib#1572, although maybe there are limitations that I'm not aware of.

[andreyvelich]

I agree with @alembiewski, it's better to have unify SDK for the training operator.
Our users just can run: pip install kubeflow-training to use it.

It seems to me that such limitations could be dropped just by regenerating the SDK models with OpenAPI generator as it was done recently for Katib: kubeflow/katib#1572, although maybe there are limitations that I'm not aware of.

Yes, we should re-generate SDK with OpenAPI to support the latest Kubernetes python client version.

[kumare3]

FYI

Ketan ping me earlier and seems Flyte user wants to use it to submit jobs. flyteorg/flyte#1375

@Jeffwan we use a custom python module to submit codes. We just reference the go api in Flyte

[Jeffwan]

kubernetes client version lock-in makes the SDK incompatible with the other Kubeflow SDKs, such as KFserving 0.5.1, KFP with the upcoming Katib SDK.

Agree. It would be hard for all projects to have same version. We can try to have same version as others.

inconsistencies between the API and SDK (e.g.: Update swagger.json schema for TFJobSpec to include RunPolicy #1278)

Yeah, I think at least we should regenerate it to reflect latest changes. I cut #1389 and it introduces some unnecessary files. We will resolve it later

[alembiewski]

@Jeffwan, any updates on this? Is there anything I can help with?