Skip to content

Conversation

hoegertn
Copy link
Contributor

@hoegertn hoegertn commented Mar 25, 2021

This PR refactors the Asset Stage of the CDK pipeline so that only one FileAsset and one DockerAsset Action are created that are uploading all assets of the respective type.

The main benefit of this is improved performance as uploading in parallel does not save enough time to compensate slower startups of multiple CodeBuilds.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Mar 25, 2021

@github-actions github-actions bot added the @aws-cdk/pipelines CDK Pipelines library label Mar 25, 2021
@mergify
Copy link
Contributor

mergify bot commented Mar 25, 2021

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

@hoegertn hoegertn changed the title WIP: feat(pipelines): restructure to only one asset upload per type WIP: feat(pipelines): restructure to only one asset upload action per type Mar 25, 2021
@hoegertn hoegertn changed the title WIP: feat(pipelines): restructure to only one asset upload action per type feat(pipelines): restructure to only one asset upload action per type Mar 25, 2021
Copy link
Contributor

@rix0rrr rix0rrr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally don't mind this change, but I had hoped it would be pluggable.

Probably this needs the AssetPublishing class to implement some common interface with two default implementations where the user can pick between, and we default to the legacy one.

@mergify mergify bot dismissed rix0rrr’s stale review April 26, 2021 21:48

Pull request has been modified.

@hoegertn
Copy link
Contributor Author

@rix0rrr I added a flag to select the publishing style

@gitpod-io
Copy link

gitpod-io bot commented May 6, 2021

@jakejscott
Copy link

jakejscott commented May 9, 2021

I'm currently running into an issue where there's too many codebuilds running at the same time to upload assets:

Action execution failed
Error calling startBuild: Cannot have more than 5 builds in queue for the account (Service: AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException; Request ID: 05ba326d-20e0-44e5-8375-1a82fc960927; Proxy: null)

I'm assuming this PR will help solve this problem so 👍

assets

@mwiarda-fette
Copy link

mwiarda-fette commented May 12, 2021

Action execution failed
Error calling startBuild: Cannot have more than 5 builds in queue for the account (Service: AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException; Request ID: 05ba326d-20e0-44e5-8375-1a82fc960927; Proxy: null)

@jakejscott did you solve your issue with codebuild? We face the same thing. Since a few days our build pipeline just refuses to work...

rix0rrr
rix0rrr previously approved these changes May 31, 2021
@rix0rrr rix0rrr self-requested a review May 31, 2021 12:34
@rix0rrr rix0rrr dismissed their stale review May 31, 2021 12:40

Request test

@rix0rrr rix0rrr changed the title feat(pipelines): restructure to only one asset upload action per type feat(pipelines): allow switching to one CodeBuild action for same-typed assets May 31, 2021
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 04a0e60
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit ed72ad3 into aws:master May 31, 2021
@mergify
Copy link
Contributor

mergify bot commented May 31, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@hoegertn hoegertn deleted the hoegertn-pipelineassets branch May 31, 2021 14:16
@jakejscott
Copy link

Thanks for the work on this, much appreciated :-)

hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
…ed assets (aws#13803)

This PR refactors the Asset Stage of the CDK pipeline so that only one FileAsset and one DockerAsset Action are created that are uploading all assets of the respective type.

The main benefit of this is improved performance as uploading in parallel does not save enough time to compensate slower startups of multiple CodeBuilds.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@HeskethGD
Copy link

HeskethGD commented Dec 4, 2022

Unless I'm missing it, the README or docs don't seem to be very explicit as to which resource the singlePublisherPerType attribute should be set on. Perhaps it could be clarified as to how to implement this.

Update: Think I found it, this appears to be called publish_assets_in_parallel now (python) https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/CodePipeline.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/pipelines CDK Pipelines library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants