Is it possible to use k3s Backup / Restore functionality in k3d?

[Sodman]

In k3s, backing up the server via the etc storage snapshot is supported, even including automatically saving that snapshot to AWS S3:

k3s etcd-snapshot \
  --s3 \
  --s3-bucket=<S3-BUCKET-NAME> \
  --s3-access-key=<S3-ACCESS-KEY> \
  --s3-secret-key=<S3-SECRET-KEY>

Resuming from that snapshot is equally easy:

k3s server \
  --cluster-init \
  --cluster-reset \
  --etcd-s3 \
  --cluster-reset-restore-path=<SNAPSHOT-NAME> \
  --etcd-s3-bucket=<S3-BUCKET-NAME> \
  --etcd-s3-access-key=<S3-ACCESS-KEY> \
  --etcd-s3-secret-key=<S3-SECRET-KEY>

Is it possible to do this, or something similar, with k3d?

My specific use case is that my CI/CD pipeline involves spinning up a local cluster and deploying a half dozen applications to it before the actual tests can run. This takes a few minutes, on every test run. If I could start the k3d cluster by restoring from a snapshot rather than setting it all up from scratch every time, it would be a significant speedup in our CI pipeline.

Thanks in advance!

[iwilltry42]

Hi @Sodman , interesting optimization!

I've never tried it, but I don't see a reason why it should not work the same way 🤔

You'd have to run k3s etcd-snapshot via docker exec k3d-<clustername>-server-0 and for the second part add all that flags using k3d's --k3s-arg flag with the @server:0 node filter. (Note: k3d adds --cluster-init automatically when you create a cluster with more than one server node.

[Sodman]

Thanks for the reply! Do I need to pass it a specific path for the etcd database to the snapshot command? When I run

docker exec k3d-test01-server-0 k3s etcd-snapshot

It's erroring out with

time="2023-03-14T12:44:50Z" level=fatal msg="etcd database not found in /var/lib/rancher/k3s/server"

This is on a fresh k3d cluster created with k3d cluster create test01 --k3s-arg "--disable=traefik@server:*"

[Sodman]

Even more puzzling is that the database does actually appear to be exactly where it's supposed to be, poking around in the container:

/ # cd /var/lib/rancher/k3s/server/
/var/lib/rancher/k3s/server # ls -la
total 36
drwx------ 8 0 0 4096 Mar 14 12:49 .
drwxr-xr-x 4 0 0 4096 Mar 14 12:49 ..
lrwxrwxrwx 1 0 0   33 Mar 14 12:49 agent-token -> /var/lib/rancher/k3s/server/token
drwx------ 2 0 0 4096 Mar 14 12:49 cred
drwx------ 3 0 0 4096 Mar 14 12:50 db
drwx------ 2 0 0 4096 Mar 14 12:49 etc
srw------- 1 0 0    0 Mar 14 12:49 kine.sock
drwx------ 3 0 0 4096 Mar 14 12:49 manifests
lrwxrwxrwx 1 0 0   33 Mar 14 12:49 node-token -> /var/lib/rancher/k3s/server/token
drwx------ 3 0 0 4096 Mar 14 12:49 static
drwx------ 4 0 0 4096 Mar 14 12:49 tls
-rw------- 1 0 0   97 Mar 14 12:49 token
/var/lib/rancher/k3s/server # cd db/
/var/lib/rancher/k3s/server/db # ls -la
total 5636
drwx------ 3 0 0    4096 Mar 14 12:50 .
drwx------ 8 0 0    4096 Mar 14 12:49 ..
drwx------ 2 0 0    4096 Mar 14 12:50 etcd
-rw-r--r-- 1 0 0 1167360 Mar 14 12:54 state.db
-rw-r--r-- 1 0 0   32768 Mar 14 12:54 state.db-shm
-rw-r--r-- 1 0 0 4548512 Mar 14 12:54 state.db-wal
/var/lib/rancher/k3s/server/db #

[Sodman]

Update, looks like the error message is thrown here which is ultimately caused by the wal dir not yet existing.

Exec'ing into the container and creating an empty member folder with an empty wal dir seems to unblock the snapshot command, or at least remove the errors... But then the command hangs on "Applying CRD helmchartconfigs.helm.cattle.io" (for > 10 mins), so I'm guessing manually creating the member + wal folders isn't a good idea.

 docker exec k3d-test03-server-0 k3s etcd-snapshot
time="2023-03-14T14:04:26Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
time="2023-03-14T14:04:26Z" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2023-03-14T14:04:26Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2023-03-14T14:04:26Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"

[Sodman]

One last update - I hadn't realized that k3s uses SQLite as the default datastore, which makes sense why the etcd backup wasn't working :)

I shifted gears and now I'm copying the /var/lib/rancher/k3s/server/db folder from a running cluster, destroying that cluster, then creating a fresh cluster with that folder pre-mounted (along with the token from the prior cluster) - and everything works now 🙌