ecr: Failed to get authorization token: not found, ResolveEndpointV2

[ashi009]

We encountered a weird error, which seems from the generated code.

time="2023-11-21T00:22:55Z" level=debug msg="Retrieving credentials" region=us-west-2 registry=NNNN serverURL=NNNN.dkr.ecr.us-west-2.amazonaws.com service=ecr
time="2023-11-21T00:22:55Z" level=debug msg="Checking file cache" registry=NNNN
time="2023-11-21T00:22:55Z" level=debug msg="Cached token is no longer valid" expiresAt="2023-11-17 07:24:04.14 +0000 UTC" requestedAt="2023-11-16 19:24:04.151845422 +0000 UTC"
time="2023-11-21T00:22:55Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=NNNN
time="2023-11-21T00:22:55Z" level=info msg="Got error fetching authorization token. Falling back to cached token." error="ecr: Failed to get authorization token: not found, ResolveEndpointV2"

We did some initial analysis on this.

1. Our version is 0.70, Git commit: cd92a7a

2. Failed to get authorization token: leads us to
   

   amazon-ecr-credential-helper/ecr-login/api/client.go

   Line 229 in cd92a7a

   return nil, fmt.Errorf("ecr: Failed to get authorization token: %w", err)

3. Given the error message, it's clear that the err is from c.ecrClient.GetAuthorizationToken,

   amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/aws-sdk-go-v2/service/ecr/api_op_GetAuthorizationToken.go

   Lines 23 to 36 in cd92a7a

   	func (c *Client) GetAuthorizationToken(ctx context.Context, params *GetAuthorizationTokenInput, optFns ...func(*Options)) (*GetAuthorizationTokenOutput, error) {
   	if params == nil {
   	params = &GetAuthorizationTokenInput{}
   	}
   	result, metadata, err := c.invokeOperation(ctx, "GetAuthorizationToken", params, optFns, c.addOperationGetAuthorizationTokenMiddlewares)
   	if err != nil {
   	return nil, err
   	}
   	out := result.(*GetAuthorizationTokenOutput)
   	out.ResultMetadata = metadata
   	return out, nil
   	}

4. Following the invocations from the c.invokeOperation,

   amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/aws-sdk-go-v2/service/ecr/api_client.go

   Lines 71 to 106 in cd92a7a

   	func (c *Client) invokeOperation(ctx context.Context, opID string, params interface{}, optFns []func(*Options), stackFns ...func(*middleware.Stack, Options) error) (result interface{}, metadata middleware.Metadata, err error) {
   	ctx = middleware.ClearStackValues(ctx)
   	stack := middleware.NewStack(opID, smithyhttp.NewStackRequest)
   	options := c.options.Copy()
   	for _, fn := range optFns {
   	fn(&options)
   	}
   	finalizeRetryMaxAttemptOptions(&options, *c)
   	finalizeClientEndpointResolverOptions(&options)
   	for _, fn := range stackFns {
   	if err := fn(stack, options); err != nil {
   	return nil, metadata, err
   	}
   	}
   	for _, fn := range options.APIOptions {
   	if err := fn(stack); err != nil {
   	return nil, metadata, err
   	}
   	}
   	handler := middleware.DecorateHandler(smithyhttp.NewClientHandler(options.HTTPClient), stack)
   	result, metadata, err = handler.Handle(ctx, params)
   	if err != nil {
   	err = &smithy.OperationError{
   	ServiceID: ServiceID,
   	OperationName: opID,
   	Err: err,
   	}
   	}
   	return result, metadata, err
   	}

   , and it calls c.addOperationGetAuthorizationTokenMiddlewares

   amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/aws-sdk-go-v2/service/ecr/api_op_GetAuthorizationToken.go

   Lines 65 to 139 in cd92a7a

   	func (c *Client) addOperationGetAuthorizationTokenMiddlewares(stack *middleware.Stack, options Options) (err error) {
   	if err := stack.Serialize.Add(&setOperationInputMiddleware{}, middleware.After); err != nil {
   	return err
   	}
   	err = stack.Serialize.Add(&awsAwsjson11_serializeOpGetAuthorizationToken{}, middleware.After)
   	if err != nil {
   	return err
   	}
   	err = stack.Deserialize.Add(&awsAwsjson11_deserializeOpGetAuthorizationToken{}, middleware.After)
   	if err != nil {
   	return err
   	}
   	if err := addProtocolFinalizerMiddlewares(stack, options, "GetAuthorizationToken"); err != nil {
   	return fmt.Errorf("add protocol finalizers: %v", err)
   	}
   	if err = addlegacyEndpointContextSetter(stack, options); err != nil {
   	return err
   	}
   	if err = addSetLoggerMiddleware(stack, options); err != nil {
   	return err
   	}
   	if err = awsmiddleware.AddClientRequestIDMiddleware(stack); err != nil {
   	return err
   	}
   	if err = smithyhttp.AddComputeContentLengthMiddleware(stack); err != nil {
   	return err
   	}
   	if err = addResolveEndpointMiddleware(stack, options); err != nil {
   	return err
   	}
   	if err = v4.AddComputePayloadSHA256Middleware(stack); err != nil {
   	return err
   	}
   	if err = addRetryMiddlewares(stack, options); err != nil {
   	return err
   	}
   	if err = awsmiddleware.AddRawResponseToMetadata(stack); err != nil {
   	return err
   	}
   	if err = awsmiddleware.AddRecordResponseTiming(stack); err != nil {
   	return err
   	}
   	if err = addClientUserAgent(stack, options); err != nil {
   	return err
   	}
   	if err = smithyhttp.AddErrorCloseResponseBodyMiddleware(stack); err != nil {
   	return err
   	}
   	if err = smithyhttp.AddCloseResponseBodyMiddleware(stack); err != nil {
   	return err
   	}
   	if err = addSetLegacyContextSigningOptionsMiddleware(stack); err != nil {
   	return err
   	}
   	if err = stack.Initialize.Add(newServiceMetadataMiddleware_opGetAuthorizationToken(options.Region), middleware.Before); err != nil {
   	return err
   	}
   	if err = awsmiddleware.AddRecursionDetection(stack); err != nil {
   	return err
   	}
   	if err = addRequestIDRetrieverMiddleware(stack); err != nil {
   	return err
   	}
   	if err = addResponseErrorMiddleware(stack); err != nil {
   	return err
   	}
   	if err = addRequestResponseLogging(stack, options); err != nil {
   	return err
   	}
   	if err = addDisableHTTPSMiddleware(stack, options); err != nil {
   	return err
   	}
   	return nil
   	}

5. There are many return points of naked errors, we need to figure out which branch gives not found, ResolveEndpointV2. It turned out "not found, %v" is from
   

   amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/smithy-go/middleware/ordered_group.go

   Lines 178 to 230 in cd92a7a

   	func (s *relativeOrder) Insert(relativeTo string, pos RelativePosition, ids ...string) error {
   	if len(ids) == 0 {
   	return nil
   	}
   	for _, id := range ids {
   	if _, ok := s.has(id); ok {
   	return fmt.Errorf("already exists, %v", id)
   	}
   	}
   	i, ok := s.has(relativeTo)
   	if !ok {
   	return fmt.Errorf("not found, %v", relativeTo)
   	}
   	return s.insert(i, pos, ids...)
   	}
   	// Swap will replace the item id with the to item. Returns an
   	// error if the original item id does not exist. Allows swapping out an
   	// item for another item with the same id.
   	func (s *relativeOrder) Swap(id, to string) error {
   	i, ok := s.has(id)
   	if !ok {
   	return fmt.Errorf("not found, %v", id)
   	}
   	if _, ok = s.has(to); ok && id != to {
   	return fmt.Errorf("already exists, %v", to)
   	}
   	s.order[i] = to
   	return nil
   	}
   	func (s *relativeOrder) Remove(id string) error {
   	i, ok := s.has(id)
   	if !ok {
   	return fmt.Errorf("not found, %v", id)
   	}
   	s.order = append(s.order[:i], s.order[i+1:]...)
   	return nil
   	}
   	func (s *relativeOrder) List() []string {
   	return s.order
   	}
   	func (s *relativeOrder) Clear() {
   	s.order = s.order[0:0]
   	}

   
   and 2 other functions in the same file.
   Search ResolveEndpointV2 gives us 3 results:

6. amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/aws-sdk-go-v2/service/ecr/api_client.go

   Lines 132 to 146 in cd92a7a

   	func addProtocolFinalizerMiddlewares(stack *middleware.Stack, options Options, operation string) error {
   	if err := stack.Finalize.Add(&resolveAuthSchemeMiddleware{operation: operation, options: options}, middleware.Before); err != nil {
   	return fmt.Errorf("add ResolveAuthScheme: %v", err)
   	}
   	if err := stack.Finalize.Insert(&getIdentityMiddleware{options: options}, "ResolveAuthScheme", middleware.After); err != nil {
   	return fmt.Errorf("add GetIdentity: %v", err)
   	}
   	if err := stack.Finalize.Insert(&resolveEndpointV2Middleware{options: options}, "GetIdentity", middleware.After); err != nil {
   	return fmt.Errorf("add ResolveEndpointV2: %v", err)
   	}
   	if err := stack.Finalize.Insert(&signRequestMiddleware{}, "ResolveEndpointV2", middleware.After); err != nil {
   	return fmt.Errorf("add Signing: %v", err)
   	}
   	return nil
   	}

7. amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/aws-sdk-go-v2/aws/signer/v4/middleware.go

   Lines 104 to 106 in cd92a7a

   	func AddUnsignedPayloadMiddleware(stack *middleware.Stack) error {
   	return stack.Finalize.Insert(&unsignedPayload{}, "ResolveEndpointV2", middleware.After)
   	}

8. amazon-ecr-credential-helper/ecr-login/vendor/github.com/aws/aws-sdk-go-v2/service/ecr/api_client.go

   Lines 451 to 455 in cd92a7a

   	func addDisableHTTPSMiddleware(stack *middleware.Stack, o Options) error {
   	return stack.Finalize.Insert(&disableHTTPSMiddleware{
   	DisableHTTPS: o.EndpointOptions.DisableHTTPS,
   	}, "ResolveEndpointV2", middleware.After)
   	}

However, this means the previously added ResolveEndpointV2 is missing after a few invocations. Could you please take a look?

[seternate]

I ran into this problem lately using Tekton. It came down that the credential-helper uses an older version of the aws-sdk and with v1.23.0 they introduced a breaking change. If you now use any depdendency using a version >v1.23.0 you will most likely run into this. See issue of aws-sdk aws/aws-sdk-go-v2#2370 and the Tekton issue tektoncd/pipeline#7698.

Hope I could help you identifing your problem a bit better.