Question around GC Roots and performance improvements

[madjam002]

Hi team, first of all just wanted to say a massive thank-you for nix-snapshotter, I've been meaning to try it out for a while now and finally had some time to play around with it, initial impressions are really positive!

I am working with a large NodeJS monorepo which I've packaged with Nix using my tool https://github.com/madjam002/yarnpnp2nix

One problem with this is some of the images I've tried creating have maybe hundreds or thousands of tiny dependencies, as with yarnpnp2nix each NPM package is a separate derivation.

Building the image and pushing to a Nix Cache (I've been testing attic) is no problem at all, even a cold push with a few thousand dependencies only takes around 15 seconds.

There does seem to be a considerable delay however in pulling the container (I'm using the nix:0/ style format), even if most of the dependencies already exist in the local /nix/store

I've narrowed this down to nix-snapshotter realising each individual package from exportReferencesGraph and also adding a GC root, and I'm wondering if there was a specific reason why this design decision was made?

I'm thinking instead it might be a good idea to just realise the top level container image and add that as a GC root, rather than each package in the closure, as that will be much quicker.

To test this idea, I patched nix-snapshotter to include the config JSON file in the image, and then only the config JSON file is realised and added as a GC root. This is a massive hack but I just wanted to prototype this idea to see if it made a difference.

You can see my patches here https://github.com/pdtpartners/nix-snapshotter/compare/main...madjam002:nix-snapshotter:single-gc-root?expand=1

Test results with an image with a few hundred dependencies:

Before patch

# time IMAGE_SERVICE_ENDPOINT=unix:///run/nix-snapshotter/nix-snapshotter.sock crictl pull nix:0/nix/store/XXX.tar
Image is up to date for sha256:XXX

real	0m6.831s # Takes almost 7 seconds
user	0m0.098s
sys	0m0.049s

After patch

# time IMAGE_SERVICE_ENDPOINT=unix:///run/nix-snapshotter/nix-snapshotter.sock crictl pull nix:0/nix/store/XXX.tar
Image is up to date for sha256:XXX

real	0m0.977s # less than 1 second
user	0m0.100s
sys	0m0.110s

The observed difference in a test K3s cluster with an image with 2000 dependencies is even more obvious, before the patch it took 120 seconds in the ContainerCreating state, now it takes 4 seconds.

Let me know what your thoughts are :)

EDIT: With nix-snapshotter and this patch I've shaved over 5 minutes off my build + deploy times! No more messing around with pushing Docker images 😃

[elpdt852]

Hi @madjam002, this is great and I'm excited to dig into this. December is generally busy for us but I'd like to review this in detail in the coming weeks.