Skip to content

Add x-azcopy and x-azcopy-sas binary caching sources #1738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

vicroms
Copy link
Member

@vicroms vicroms commented Jul 29, 2025

Adds two new binary sources that interact with Azure Storage blob containers using the azcopy executable.

  • x-azcopy-sas,<url>,<sas>[,<rw>]: Intended as a drop-in replacement for azblob, it uses SAS tokens for authentication.
  • x-azcopy,<url>[,<rw>]: Intended for non-SAS authentication methods like Microsoft Entra ID; requires preloading credentials or/and the AZCOPY_AUTO_LOGIN_TYPE.

Internally, we use azcopy copy --from-to BlobLocal {destination} --include-path {files} argument to download multiple files in a single azcopy job, this improves performance and brings it more in line with azblob, that uses a single curl invocation to download a batch of files. Upload happens one file at a time in the background.

Small test

x-azblob:
    Restored 5 package(s) from HTTP servers in 3.8 s. Use --debug to see more details.

x-azcopy:
    Restored 5 package(s) from Azure Storage in 1.5 min. Use --debug to see more details.

x-azcopy-sas:
    Restored 5 package(s) from Azure Storage in 4.7 s. Use --debug to see more details.

Large test

azcopy-sas (with --include-path):
    Restored 141 package(s) from Azure Storage in 1.5 min. Use --debug to see more details.

azblob:
    Restored 141 package(s) from HTTP servers in 3 min. Use --debug to see more details.

Related:

@vicroms vicroms changed the title Add x-azcopy provider Add x-azcopy and x-azcopy-sas binary caching sources Aug 1, 2025
@vicroms vicroms marked this pull request as ready for review August 1, 2025 18:43
@vicroms vicroms marked this pull request as draft August 5, 2025 19:24
@vicroms
Copy link
Member Author

vicroms commented Aug 5, 2025

@vicroms vicroms marked this pull request as ready for review August 7, 2025 19:19
Copy link
Member

@BillyONeal BillyONeal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only comment I'm 'request changes' over is the one on line 1078 where the for loop seems clearly broken. The other things are nitpicks.

BillyONeal
BillyONeal previously approved these changes Aug 12, 2025
@ras0219-msft ras0219-msft requested a review from Copilot August 12, 2025 22:10
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds two new binary caching sources for vcpkg that use Azure Storage blob containers via the azcopy executable. The implementation provides both SAS token-based authentication and Microsoft Entra ID authentication options.

Key changes:

  • Introduces x-azcopy and x-azcopy-sas binary cache providers as alternatives to existing Azure blob storage options
  • Implements batch downloading using azcopy copy --include-path for improved performance compared to individual file downloads
  • Adds comprehensive test coverage for both new provider types with various configuration scenarios

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/vcpkg/metrics.cpp Adds metrics tracking for the two new azcopy binary cache providers
src/vcpkg/binarycaching.cpp Core implementation of AzCopy providers with batch download/upload logic and configuration parsing
src/vcpkg-test/configparser.cpp Comprehensive unit tests for both azcopy provider variants with various configuration scenarios
locales/messages.json Adds localized message for Azure Storage restoration feedback
include/vcpkg/metrics.h Defines new metric types for tracking azcopy provider usage
include/vcpkg/binarycaching.h Declares AzCopyUrl structure and adds provider state fields
include/vcpkg/base/message-data.inc.h Declares the new Azure Storage restoration message

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants