feat(Backend + SDK): Update kfp backend and kubernetes sdk to support tolerations #10471

droctothorpe · 2024-02-12T18:06:14Z

Description of your changes:
This PR adds support for tolerations to the driver and kubernetes_platform SDK.

It is a follow up to #10442, which updated the protobuf and Golang stub.

It follows the excellent example set by #10427.

Huge thanks to @Tomcli for paving the way and sharing how to validate things end to end.

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

google-oss-prow · 2024-02-12T18:06:25Z

Hi @droctothorpe. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubernetes_platform/proto/kubernetes_executor_config.proto

backend/src/v2/driver/driver_test.go

backend/src/v2/driver/driver.go

kubernetes_platform/python/test/snapshot/data/toleration.py

kubernetes_platform/python/kfp/kubernetes/__init__.py

kubernetes_platform/python/test/unit/test_tolerations.py

droctothorpe · 2024-02-13T18:02:30Z

Rebased and resolved merge conflicts.

connor-mccarthy

Thanks, @droctothorpe!

kubernetes_platform/python/kfp/kubernetes/__init__.py

kubernetes_platform/python/kfp/kubernetes/toleration.py

kubernetes_platform/proto/kubernetes_executor_config.proto

kubernetes_platform/python/kfp/kubernetes/toleration.py

droctothorpe · 2024-02-14T19:01:28Z

@connor-mccarthy We addressed all feedback. We put it in a discrete commit to make review easier, but happy to squash if that's helpful. Thank you! 🙏

Tomcli · 2024-02-14T22:54:33Z

/ok-to-test

Tomcli · 2024-02-14T22:56:57Z

fyi @droctothorpe your tests will be failing because the backend is pulling the kubernetes_platform package from github upstream instead of local packages. This is why the PRs our team created have to at least separate into two PRs.

droctothorpe · 2024-02-14T23:16:58Z

fyi @droctothorpe your tests will be failing because the backend is pulling the kubernetes_platform package from github upstream instead of local packages. This is why the PRs our team created have to at least separate into two PRs.

Doh. That makes sense. I'll rebase and resolve merge conflicts in the other PR this evening. Thanks, @Tomcli!

droctothorpe · 2024-02-14T23:54:37Z

@Tomcli I rebased and re-opened #10442. It should be good to go. Does it make sense to add the kubernetes_platform package to the replace directive either in general or for CI specifically? That would negate the need to submit these kinds of changes in discrete PRs. Just a thought.

kubernetes_platform/python/kfp/kubernetes/toleration.py

Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]>

connor-mccarthy

/lgtm
/approve

Thank you, @droctothorpe!

droctothorpe · 2024-02-20T23:21:51Z

/lgtm /approve

Thank you, @droctothorpe!

Still trying to debug why the license checks are failing in CI.

droctothorpe · 2024-02-21T01:14:33Z

/retest-required

droctothorpe · 2024-02-21T03:16:02Z

All required tests are finally passing. Thanks for your patience! CC @connor-mccarthy. Just gonna squash the last few commits to get past the DCO gate.

Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]>

google-oss-prow · 2024-02-21T04:10:29Z

@droctothorpe: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
kubeflow-pipelines-samples-v2	`dda1ec3`	link	false	`/test kubeflow-pipelines-samples-v2`
kfp-kubernetes-execution-tests	`dda1ec3`	link	false	`/test kfp-kubernetes-execution-tests`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Tomcli

thanks @droctothorpe
/lgtm

google-oss-prow · 2024-02-22T19:42:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun, connor-mccarthy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [chensun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

… tolerations (kubeflow#10471) * feat(Backend + SDK): Update kfp backend and kubernetes sdk to support tolerations Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]> * Address PR review 1 Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]> * Refactor add_toleration to use Python primitives Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]> * Update go.mod to pull in latest kubernetes_platform package Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]> --------- Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]>

google-oss-prow bot requested review from chensun and gkcalat February 12, 2024 18:06

google-oss-prow bot added the size/XL label Feb 12, 2024

google-oss-prow bot added the needs-ok-to-test label Feb 12, 2024

droctothorpe force-pushed the tolerations2 branch from a382a77 to 002bfe9 Compare February 12, 2024 18:11

droctothorpe commented Feb 12, 2024

View reviewed changes

kubernetes_platform/proto/kubernetes_executor_config.proto Show resolved Hide resolved

edmondop suggested changes Feb 12, 2024

View reviewed changes

droctothorpe force-pushed the tolerations2 branch from 002bfe9 to 3adeda4 Compare February 12, 2024 19:33

nithin8702 mentioned this pull request Feb 13, 2024

chore(sdk): release KFP SDK 2.7.0 #10461

Merged

1 task

droctothorpe force-pushed the tolerations2 branch from 3adeda4 to cad6929 Compare February 13, 2024 17:59

connor-mccarthy requested changes Feb 13, 2024

View reviewed changes

droctothorpe force-pushed the tolerations2 branch from 6c5fe9d to 9036732 Compare February 14, 2024 18:59

droctothorpe mentioned this pull request Feb 14, 2024

feat(kubernetes_platform): Update kubernetes_platform go package to i… #10442

Merged

1 task

droctothorpe requested a review from connor-mccarthy February 14, 2024 20:11

google-oss-prow bot added ok-to-test and removed needs-ok-to-test labels Feb 14, 2024

zijianjoy assigned connor-mccarthy Feb 15, 2024

droctothorpe force-pushed the tolerations2 branch from 9036732 to 91affce Compare February 16, 2024 00:10

Tomcli reviewed Feb 16, 2024

View reviewed changes

kubernetes_platform/python/kfp/kubernetes/toleration.py Outdated Show resolved Hide resolved

droctothorpe force-pushed the tolerations2 branch from 91affce to 7e8cff9 Compare February 16, 2024 21:14

google-oss-prow bot added size/L and removed size/XL labels Feb 16, 2024

droctothorpe commented Feb 16, 2024

View reviewed changes

kubernetes_platform/python/kfp/kubernetes/toleration.py Show resolved Hide resolved

droctothorpe and others added 2 commits February 20, 2024 17:42

Address PR review 1

49c906b

Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]>

Refactor add_toleration to use Python primitives

bfa24d5

Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]>

connor-mccarthy approved these changes Feb 20, 2024

View reviewed changes

google-oss-prow bot added the lgtm label Feb 20, 2024

droctothorpe force-pushed the tolerations2 branch from ab08ffb to aadbb5a Compare February 20, 2024 23:18

google-oss-prow bot removed the lgtm label Feb 20, 2024

droctothorpe force-pushed the tolerations2 branch from aadbb5a to 2339989 Compare February 20, 2024 23:19

droctothorpe force-pushed the tolerations2 branch 2 times, most recently from a6ba900 to 6819b40 Compare February 21, 2024 00:40

Update go.mod to pull in latest kubernetes_platform package

dda1ec3

Signed-off-by: droctothorpe <[email protected]> Co-authored-by: edmondop <[email protected]> Co-authored-by: tarat44 <[email protected]>

droctothorpe force-pushed the tolerations2 branch from a7f6185 to dda1ec3 Compare February 21, 2024 03:18

Tomcli reviewed Feb 21, 2024

View reviewed changes

google-oss-prow bot assigned Tomcli Feb 21, 2024

google-oss-prow bot added the lgtm label Feb 21, 2024

chensun approved these changes Feb 22, 2024

View reviewed changes

google-oss-prow bot added the approved label Feb 22, 2024

google-oss-prow bot merged commit 2983a7d into kubeflow:master Feb 22, 2024

droctothorpe deleted the tolerations2 branch February 22, 2024 22:04

This was referenced Feb 27, 2024

REQUEST: New membership for droctothorpe kubeflow/internal-acls#656

Closed

Add droctothorpe as member kubeflow/internal-acls#657

Merged

droctothorpe mentioned this pull request Mar 17, 2024

[feature] Add ability to specify node affinity & toleration using KFP V2 #9682

Closed

droctothorpe mentioned this pull request Jun 5, 2024

feat(Backend + SDK): Update kfp backend and kubernetes sdk to support node affinity #10672

Closed

1 task

feat(Backend + SDK): Update kfp backend and kubernetes sdk to support tolerations #10471

feat(Backend + SDK): Update kfp backend and kubernetes sdk to support tolerations #10471

Uh oh!

Conversation

droctothorpe commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-oss-prow bot commented Feb 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

droctothorpe commented Feb 13, 2024

Uh oh!

connor-mccarthy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

droctothorpe commented Feb 14, 2024

Uh oh!

Tomcli commented Feb 14, 2024

Uh oh!

Tomcli commented Feb 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

droctothorpe commented Feb 14, 2024

Uh oh!

droctothorpe commented Feb 14, 2024

Uh oh!

Uh oh!

Uh oh!

connor-mccarthy left a comment

Choose a reason for hiding this comment

Uh oh!

droctothorpe commented Feb 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

droctothorpe commented Feb 21, 2024

Uh oh!

droctothorpe commented Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-oss-prow bot commented Feb 21, 2024

Uh oh!

Tomcli left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Feb 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

droctothorpe commented Feb 12, 2024 •

edited

Loading

Tomcli commented Feb 14, 2024 •

edited

Loading

droctothorpe commented Feb 20, 2024 •

edited

Loading

droctothorpe commented Feb 21, 2024 •

edited

Loading