Skip to content

Conversation

@gargipanatula
Copy link
Contributor

@gargipanatula gargipanatula commented May 7, 2025

What type of PR is this?

Uncomment only one, leave it on its own line:

/kind api-change
/kind bug

/kind cleanup

/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
The AWS SDK Go V1 is being deprecated on July 31, 2025. This PR migrates usage of the EC2 SDK to use V2.
Context: https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-go-v1-on-july-31-2025/

Changes:

  • CredentialsChainVerboseErrors has been removed from the EC2 Client (see NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors aws/aws-sdk-go#2914)
  • In v1, BeforeSign caused some requests to be marked as non-retryable in some cases. Now, due to the V2's migration to middleware, the Finalize step (through delayPreSign()) now throws errors with a custom nonRetryableError error code when the same conditions are met. The EC2 Client is configured with a custom retrier, customRetryer, which will catch this error and cause the request to not be retried.
  • Logging: AWS API Send has been split up into two logs due to changes in where the logged information is available throughout the middleware. Request parameters are only available in the Initialize and Serialize step.
    Old logs:
newer

I0514 05:05:54.252687      11 log_handler.go:27] AWS request: ec2 CreateTags
  
I0514 05:05:54.452106      11 log_handler.go:32] AWS API Send: ec2 CreateTags &{CreateTags POST / <nil> <nil>} {
  Resources: ["i-09271f1185db6f280"],
  Tags: [{
      Key: "aws:eks:cluster-name",
      Value: "panatula513116"
    }]
}

I0514 05:05:54.452139      11 log_handler.go:37] AWS API ValidateResponse: ec2 CreateTags &{CreateTags POST / <nil> <nil>} {
  Resources: ["i-09271f1185db6f280"],
  Tags: [{
      Key: "aws:eks:cluster-name",
      Value: "panatula513116"
    }]
} 200 OK

older

New logs:

newer

AWS API Send: EC2 CreateTags &{[i-0adbe1d0ed4bb2329] [{0xc000a20e60 0xc000a20e70 {}}] <nil> {}}

AWS API Send: EC2 CreateTags POST /

AWS request: EC2 CreateTags

AWS API ValidateResponse: EC2 CreateTags 200

older

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Testing details:
Added unit tests to verify custom endpoint resolver and retrier in the EC2 Client, included in the aws_sdk_test.go file.
Ran make && make test. Created a personal cluster and manually added nodes to verify that tags were still properly added. Ran e2e tests on the cluster using the following commands:

export KUBECONFIG=<kubeconfig path>
export AWS_REGION=us-west-2
export TEST_PATH=./tests/e2e/...
export GINKGO_NODES=4
export GINKGO_FOCUS=[cloud-provider-aws-e2e]
export GINGKO_SKIP=[Disruptive]
go install github.com/onsi/ginkgo/[email protected]
export PATH=$PATH:$HOME/go/bin
pushd ./tests/e2e
ginkgo . -v -p --nodes=$GINKGO_NODES --focus=$GINKGO_FOCUS --skip=$GINKGO_SKIP

<...test output...>
Ran 3 of 3 Specs in 72.299 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestE2E (72.30s)
PASS

@k8s-ci-robot k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 7, 2025
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented May 7, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 7, 2025
@k8s-ci-robot k8s-ci-robot requested review from dims and olemarkus May 7, 2025 19:26
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot
Copy link
Contributor

Welcome @gargipanatula!

It looks like this is your first PR to kubernetes/cloud-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

Hi @gargipanatula. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 7, 2025
@gargipanatula gargipanatula changed the title Upgrade AWS EC2 to Go SDK V1 Upgrade AWS EC2 to Go SDK V2 May 7, 2025
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 7, 2025
ec2.InstanceStateNameShuttingDown,
ec2.InstanceStateNameStopping,
ec2.InstanceStateNameStopped,
string(ec2types.InstanceStateNamePending),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: change the array type []string{..} to the new ec2 type and cast it to string when needed

Copy link
Contributor Author

@gargipanatula gargipanatula May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that this would be better, but it's only used as a parameter to newEc2Filter, which expects a []string{}. Converting aliveFilter to strings just for that function might be a little messy, so maybe we can keep it as a []string{}? Let me know if that sounds ok, open to either way.


// An interface to satisfy the ec2.Client API.
// More details about this pattern: https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/unit-testing.html
type EC2API interface {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this interface type? is it for testing purpose?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Previously, awsSdkEC2 and MockedEC2API wrapped ec2iface.EC2API to mock the ec2 client. In v2, the API doesn't exist, so we have to create our own interface and wrap that instead.
More details: https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/unit-testing.html

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we can only keep the interface for functions required for testing for simplicity. For example, if only CreateSecurityGroup is needed for the tests, we can only keep it and remove others.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that makes sense. All these guys are mocked though, mostly by FakeEC2Impl, so maybe we can keep it as is for now to minimize changes in this PR and address it in the future.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename it to EC2APIForTesting?

return kmsClient, nil
}

func (p *awsSDKProvider) AddHandlersV2(ctx context.Context, regionName string, cfg awsv2.Config) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a code comment here to note that original implementation in SDK v1?

type EC2 interface {
// Query EC2 for instances matching the filter
DescribeInstances(request *ec2.DescribeInstancesInput) ([]*ec2.Instance, error)
DescribeInstances(ctx context.Context, request *ec2.DescribeInstancesInput, optFns ...func(*ec2.Options)) ([]ec2types.Instance, error)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need EC2API interface above, if we already have this one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EC2API is used to mock the EC2 client in general, whereas this interface is used to provide custom implementations over existing EC2 functionality (e.g. returning []*ec2.SecurityGroup instead of the default ec2.DescribeSecurityGroupsOutput in DescribeSecurityGroups()). I assume this was originally created for mocking, because some operations return paginated results which are hard to mock.

One example of how this is in findSecurityGroup. Cloud.ec2 is iface.EC2, and the result of cloud.ec2.DescribeSecurityGroups is []*ec2.SecurityGroup.

Compare this to DescribeSecurityGroups, where awsSdkEC2.ec2 is EC2API, and the result of awsSdkEC2.ec2.DescribeSecurityGroups is *ec2.DescribeSecurityGroupsOutput.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get it, the detailed implementation for mock can be different but shouldn't the interface be the same? We can only keep this interface and reuse it for mocking

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's because the original authors wanted two levels of mocking. The first one is that they wanted to mock the EC2 Client functions, hence their usage of ec2iface.EC2API (now EC2API). The second one is that they had their own set of functions, and to mock these, they made iface.EC2. These functions are not really related to the EC2 Client, they're essentially functions that happen to call the EC2 API and do their own thing.

I believe that if we take out iface.EC2, then we don't have a way to mock things like awsSdkEC2.DescribeSecurityGroups

return out, metadata, err
}

func awsServiceAndNameV2(ctx context.Context) (string, string) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you share the original implementation for this in v1?

klog.Infof("Starting the tagging controller")
for i := 0; i < tc.workerCount; i++ {
go wait.Until(tc.work, tc.nodeMonitorPeriod, stopCh)
go wait.Until(func() { tc.work(ctx) }, tc.nodeMonitorPeriod, stopCh)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why wrap in a func ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't wrap, it interprets tc.work(ctx) as a function call, so I wrapped it to satisfy the fact that .Until needs a func()

}

go taggingcontroller.Run(ctx.Done())
go taggingcontroller.Run(ctx, ctx.Done())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just pass ctx as the argument as the second one is based on ctx anyway?


request := &ec2.DescribeInstancesInput{
InstanceIds: []*string{instanceID.awsString()},
InstanceIds: []string{},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the instance if removed? without the filter it will query all the instances right ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad - fixed.

})

func (p *awsSDKProvider) Compute(ctx context.Context, regionName string) (iface.EC2, error) {
cfg, err := awsConfig.LoadDefaultConfig(context.TODO(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ctx is already provided as function param right, can we use it instead of TODO()?

Credentials: p.creds,
}
awsConfig = awsConfig.WithCredentialsChainVerboseErrors(true).
WithEndpointResolver(p.cfg.GetResolver())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you - added client level overrides!

}

func (p *awsSDKProvider) AddHandlersV2(ctx context.Context, regionName string, cfg awsv2.Config) {
cfg.APIOptions = append(cfg.APIOptions,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a way to test these in unit test? if not can you share the log details just to make sure we the behavior is same?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep will paste the logs in the PR description, thank you.


delayer := p.getCrossRequestRetryDelay(regionName)
if delayer != nil {
cfg.APIOptions = append(cfg.APIOptions,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we test this using an unit test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarification, do we want to add a test for getCrossRequestRetryDelay or for the middleware functions themselves? The functions called in middleware, like ComputeDelayForRequest are tested in separate unit tests.

}
func (r customRetryer) IsErrorRetryable(err error) bool {
var ae smithy.APIError
if errors.As(err, &ae) && strings.Contains(ae.Error(), NON_RETRYABLE_ERROR) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i see you are creating this error as a normal one using errors.new, then will errors.As succeed here ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you're right - errors.As would be false for NON_RETRYABLE_ERRORs. Will do a simple error message check instead.

Comment on lines -1394 to +1395
func (c *Cloud) addSecurityGroupIngress(securityGroupID string, addPermissions []*ec2.IpPermission) (bool, error) {
func (c *Cloud) addSecurityGroupIngress(ctx context.Context, securityGroupID string, addPermissions []ec2types.IpPermission) (bool, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we pass the value instead of pointer type here?

Comment on lines -1455 to +1456
func (c *Cloud) removeSecurityGroupIngress(securityGroupID string, removePermissions []*ec2.IpPermission) (bool, error) {
func (c *Cloud) removeSecurityGroupIngress(ctx context.Context, securityGroupID string, removePermissions []ec2types.IpPermission) (bool, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Comment on lines -1481 to +1485
var found *ec2.IpPermission
var found *ec2types.IpPermission
for _, groupPermission := range group.IpPermissions {
if ipPermissionExists(removePermission, groupPermission, hasUserID) {
found = removePermission
if ipPermissionExists(&removePermission, &groupPermission, hasUserID) {
found = &removePermission
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if passing value is desired for this function, we can directly use value type var found ec2types.IpPermission. There is no need to transform to pointer and back then.


existing := subnetsByAZ[az]
if existing == nil {
existing, exists := subnetsByAZ[az]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe worth thinking a new name for existing


// An interface to satisfy the ec2.Client API.
// More details about this pattern: https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/unit-testing.html
type EC2API interface {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we can only keep the interface for functions required for testing for simplicity. For example, if only CreateSecurityGroup is needed for the tests, we can only keep it and remove others.

Comment on lines 135 to 137
cfg, err := awsConfig.LoadDefaultConfig(context.TODO(),
awsConfig.WithRegion(regionName),
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When creating the config, can we use

config.LoadDefaultConfig(ctx, config.WithDefaultsMode(aws.DefaultsModeInRegion), 
  awsConfig.WithRegion(regionName),
  ... // plus any other configurations
)

that will give us default configurations https://github.com/aws/aws-sdk-go-v2/blob/main/aws/defaults/defaults.go#L26-L32

Comment on lines 141 to 142
cfg.Region = regionName
cfg.Credentials = p.credsV2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we can embed them into awsConfig.Withxxx in awsConfig.LoadDefaultConfig above

Comment on lines +147 to +149
o.Retryer = &customRetryer{
retry.NewStandard(),
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: same for retry, which can also be embedded into awsConfig.WithRetryer. But given we specify the default mode, the standard retryer will be picked by default so we don't need to explicitly set this.

If there is a need to set up custom retryer we can do something like:

awsConfig.WithRetryer(func() aws.Retryer {
    return retry.AddWithMaxAttempts(retry.NewStandard(), 5)
})


Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we need to have a custom retryer is because we need to mark requests as non retryable under specific conditions. In v2, we do this via a combination of middleware and custom retry logic (see here: https://github.com/kubernetes/cloud-provider-aws/pull/1146/files#r2082328615).

The custom retryer you shared would definitely limit the number of retries, but it would do it for all requests. In this case, we need to be able to catch a specific case and never retry that.

type EC2 interface {
// Query EC2 for instances matching the filter
DescribeInstances(request *ec2.DescribeInstancesInput) ([]*ec2.Instance, error)
DescribeInstances(ctx context.Context, request *ec2.DescribeInstancesInput, optFns ...func(*ec2.Options)) ([]ec2types.Instance, error)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get it, the detailed implementation for mock can be different but shouldn't the interface be the same? We can only keep this interface and reuse it for mocking

Comment on lines 185 to 197
// Standard retry implementation, except that it doesn't retry RequestLimitExceeded errors.
// This works in tandem with CrossRequestRetryDelay, which reports these errors before
// the middleware exits upon seeing this error
type customRetryer struct {
awsv2.Retryer
}
func (r customRetryer) IsErrorRetryable(err error) bool {
var ae smithy.APIError
if errors.As(err, &ae) && strings.Contains(ae.Error(), NON_RETRYABLE_ERROR) {
return false
}
return r.Retryer.IsErrorRetryable(err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? I didn't see it used anywhere.

v2 retryer itself has an implementation to decide which errors are retryable https://github.com/aws/aws-sdk-go-v2/blob/main/aws/retry/retryable_error.go#L15. But I didn't see the purpose of custom retryer here

Copy link
Contributor Author

@gargipanatula gargipanatula May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're replicating a handler function, which used to mark a request as non retryable. In v2, we can't directly mark a request as non retryable, so we have to 1) throw a custom error from the middleware stage and 2) have a retryer that can catch that error and return false for IsErrorRetryable().
Original handler:

func (c *CrossRequestRetryDelay) BeforeSign(r *request.Request) {

It's used for the EC2 aws sdk go v2 client:

o.Retryer = &customRetryer{

@cartermckinnon
Copy link
Contributor

/ok-to-test

@gargipanatula
Copy link
Contributor Author

/retest
/retest-required

@kmala
Copy link
Member

kmala commented May 28, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 28, 2025
@kmala
Copy link
Member

kmala commented May 28, 2025

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kmala, yue9944882

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 28, 2025
@k8s-ci-robot k8s-ci-robot merged commit 6dcae3c into kubernetes:master May 28, 2025
11 checks passed
k8s-ci-robot added a commit that referenced this pull request Jun 30, 2025
…#1146-#1157-#1169-#1177-upstream-release-1.33

Automated cherry pick of #1146: upgraded to ec2 v2, buggy
#1157: Update ELB and ELBV2 packages to AWS SDK Go V2
#1169: aws sdk go upgrade
#1177: update aws & awserr to go sdk v2
k8s-ci-robot added a commit that referenced this pull request Jul 3, 2025
…#1146-#1157-#1169-#1177-upstream-release-1.32

Automated cherry pick of #1146: upgraded to ec2 v2, buggy
#1157: Update ELB and ELBV2 packages to AWS SDK Go V2
#1169: aws sdk go upgrade
#1177: update aws & awserr to go sdk v2
k8s-ci-robot added a commit that referenced this pull request Oct 10, 2025
#1157-#1169-#1177-upstream-release-1.30

Automated cherry pick of #1146: upgraded to ec2 v2, buggy #1157: Update ELB and ELBV2 packages to AWS SDK Go V2 #1169: aws sdk go upgrade #1177: update aws & awserr to go sdk v2
k8s-ci-robot added a commit that referenced this pull request Oct 10, 2025
#1157-#1169-#1177-upstream-release-1.31

Automated cherry pick of #1146: upgraded to ec2 v2, buggy #1157: Update ELB and ELBV2 packages to AWS SDK Go V2 #1169: aws sdk go upgrade #1177: update aws & awserr to go sdk v2
k8s-ci-robot added a commit that referenced this pull request Nov 12, 2025
(1.29) Automated cherry pick of #1146: upgraded to ec2 v2, buggy #1157: Update ELB and ELBV2 packages to AWS SDK Go V2 #1169: aws sdk go upgrade #1177: update aws & awserr to go sdk v2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants