Add aws-sigv4 auth option to filetransfer and deprecate or delete s3-binary-cache-store and aws-cpp-sdk-s3

[arianvp]

Is your feature request related to a problem?

The s3-binary-cache-store (especially substitution) is extremely buggy. Meanwhile our http substituter is not buggy and way more battle-tested

• Nix binary cache, S3, InvalidChunkSizeError #12671
• S3 cache does not provide realisations endpoint for content addressed derivations  #11748
• copying from s3 cache buffers entire download in memory, causes segfault #12403

Proposed solution

Use http-binary-cache-store to talk to S3 directly

libcurl has aws-sigv4 authentication built in these days: https://curl.se/libcurl/c/CURLOPT_AWS_SIGV4.html

So we can simple use the existing FileTransfer implementation to push to and pull from S3. As S3 is simply REST semantics that map to what http-binary-cache-store already does

The only thing that we need to keep is the AWS credential chain to actually fetch the credentials to pass to curl but for that we only need to depend on https://github.com/awslabs/aws-crt-cpp or even smaller https://github.com/awslabs/aws-c-auth

This also solves the problem of people complaining that we link against aws-cpp-sdk as aws-c-auth is a way lighter dependency

aws-c-auth suffers from the same problem as #5947 but now we only need one library to enable BYO_CRYPTO instead of a whole bunch of them. So it makes things significantly easier.

Something like this in filetransfer should work. We already special case s3:// URLs in FileTransfer so we can use that to do the following instead of calling out to the S3 SDK:

// making a request to   s3.${region}.amazonaws.com/${bucket}/${key}

// TODO: get awsAccessKeyId and friends from aws-c-auth 

curl_easy_setopt(req,  CURLOPT_HTTPAUTH, CURLAUTH_AWS_SIGV4);
curl_easy_setopt(req,  CURLOPT_USERNAME, awsAccessKeyId);
curl_easy_setopt(req,  CURLOPT_PASSWORD, awsSecretKey);

if (awsSessionToken) {
  struct curl_slist *list = NULL;
  curl_slist_append(list, "x-amz-security-token", awsSessionToken)
  curl_easy_setopt(req, CURLOPT_HTTPHEADER, list);
}

Now all the HTTP PUT/GET/POST/GET operations should work as expected.

Alternative solutions

Fix all the weird bugs with the current S3 substituter

Additional context

Checklist

• checked latest Nix manual (source)
• checked open feature issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[Mic92]

Sounds like a good idea.