copying from s3 cache buffers entire download in memory, causes segfault

[andrewhamon]

Describe the bug

When copying a large path from an s3-based cache, the nix binary appears to buffer the entire download into memory.

When the path is large enough (above approximately 3.5GB) this also reliably causes nix to segfault.

I can only reproduce for an s3-based nix cache. If I use an HTTP cache, memory usage stays low and constant.

Steps To Reproduce

cd $(mktemp -d)
dd if=/dev/urandom of=./random_4g.bin bs=1M count=4096
path=$(nix store add-path . --name large-random)

# copy path to s3 store
nix copy --from local --to <the s3 cache> $path

# delete it from the local store
nix store delete $path

# attempt to copy from s3 store
nix copy --to local --from <the s3 cache> $path

# experience segfault
# 74861 segmentation fault  nix copy --to local /nix/store/rv559vmhs7751xizmfnxk5bwyjhfizpa-large-random

Expected behavior

Nix uses fixed amount of memory and does not segfault.

Metadata

nix-env (Nix) 2.22.0

but I have also experienced this with nix 2.25.0

Additional context

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[arianvp]

There is a FIXME in the code for this :/

nix/src/libstore/s3-binary-cache-store.cc

Line 427 in fc83c6c

// FIXME: stream output to sink.

s3.getObject indeed seems to buffer the entire request into memory:

nix/src/libstore/s3-binary-cache-store.cc

Line 159 in fc83c6c

S3Helper::FileTransferResult S3Helper::getObject(

We should instead use the S3 TransferManager for streamed get https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html

We're already using the TransferManager for uploads; but not for downloads :/

[arianvp]

Other option:

lib curl has support for AWS Authentication built in these days. We just have to implement the AWS credential chain and we drop the AWS-CPP-SDK dependency and just do S3 upload and download with http-binary-cache-store's libcurl-backed implementation

[arianvp]

Seems for uploading we also only use TransferManager when multipart-upload=true which is completely undocumented and defaults to false

nix/src/libstore/s3-binary-cache-store.hh

Line 87 in fc83c6c

const Setting<bool> multipartUpload{this, false, "multipart-upload", "Whether to use multi-part uploads."};

[andrewhamon]

That is an interesting option. Seems kinda high effort. Is AWS-CPP-SDK something nix is eager to drop?

[arianvp]

aws-cpp-sdk has historically not been the most nice of dependencies to manage. I'm not involved in nix day to day development but personally I'd love for the dependency to be softer. Just for the fact that nix then doesn't link against 3 different TLS libraries

[guskovd]

I can confirm this behavior.
On machines with little RAM we simply can't download large packages from s3.
Instead we have to disable substituters for local build and this is very inconvenient

[arianvp]

There's a PR that seems to address this: #6824