Use Filestore over NFS to speed up and share GCS Object Storage cached slabs

[djeebus]

Early tests show that this can reduce the startup time for sandboxes by ~60% if the sandbox is starting on a fresh node.

Notes:

• this only stores template slabs in NFS, in order to reduce disk usage and blast radius
• the launch darkly flags use-nfs-for-snapshots and use-nfs-for-templates control whether we write slabs to NFS for snapshots and templates.

Remaining tasks:

• Bring back terraform, make sure it is production ready
• Remove unused files from the NFS mount when appropriate (in progress)
• Monitor usage
• Protect against short writes to GCP Object Storage during batched writes

Other items worth noting:

1. the "build-packages" github action now calculates its cache key based on go.work/go.work.sum, since that collects all dependencies and is simpler than recursively searching the whole repository
2. there is a new tfvar called "use_filestore_cache" that enables the creation of the filestore (defaults to false)
3. make plan-only-jobs no longer errors if a change needs to be made, allowing make plan-only-jobs && make apply
4. make test now tests every package in the go.work file, rather than requiring them to be listed out one at a time
5. make lint can now be run to lint all the packages
6. I fixed a race condition in the lifecycle_cache_test.go, revealed when running the tests with -race
7. I refactored the boolean flags to combine key and default into a single struct. If we like it, we can refactor the int flags similarly.

[ValentaTomas]

Also not sure if https://github.com/e2b-dev/infra/blob/nfs-file-cache/packages/orchestrator/internal/server/sandboxes.go#L423 will the snapshot upload use the cached persistence and therefore the mentioned writes.

[djeebus]

Also not sure if https://github.com/e2b-dev/infra/blob/nfs-file-cache/packages/orchestrator/internal/server/sandboxes.go#L423 will the snapshot upload use the cached persistence and therefore the mentioned writes.

No, it's only used in the GetTemplate function, which is called on Create (line 52 in the same file). The wrapped persistence value only gets used inside GetTemplate and in the Template instance, which isn't used to upload the snapshot.

[ValentaTomas]

Also not sure if https://github.com/e2b-dev/infra/blob/nfs-file-cache/packages/orchestrator/internal/server/sandboxes.go#L423 will the snapshot upload use the cached persistence and therefore the mentioned writes.

No, it's only used in the GetTemplate function, which is called on Create (line 52 in the same file). The wrapped persistence value only gets used inside GetTemplate and in the Template instance, which isn't used to upload the snapshot.

https://github.com/e2b-dev/infra/pull/955/files/f159ab38864c36203da88527d38221bca8f07c36#diff-39e76d47dde3cef11269255fc539b2b574991b5e59265204deb19d94235ca571L75

Isn't this also using the GetTemplate?

[djeebus]

Yup, it does. The surrounding code seems to use it to pull a layer from GCS, which may cause it to write some cache files to NFS. it might also read from NFS, which might speed up read operations. I'm not sure how likely that scenario is though. It doesn't look like it ever uses the persistence field of the storageTemplate struct for anything other than reads from GCS though. I also ran e2b template build locally and verified via metrics that it doesn't cause any writes to NFS.