[Kairos Factory] Ubuntu 24.04 Custom Image Will Not Start Pods

[Itxaka]

Discussed in #3364

^{Originally posted by sdwilsh April 25, 2025}
I have a very simple setup currently, where I'm making no customization; just using what appears to be the same as the builder in this repository. When I updated to this image from an existing Kairos image on a worker node, I found that k3s came up, and connected to the control plane, but that no Pods would start. From the logs, it looked like flannel wasn't working and that the CNI couldn't be found.

Is there some additional processing that happens that isn't yet documented that might be the cause of this? Alternatively, any ideas on what I should look into to further debug this?

[sdwilsh]

Responding to some questions from the original discussion:

can you tell me what version did you upgrade from?

I upgraded from 24.04-standard-amd64-generic-v3.3.1-k3sv1.32.1-k3s1. I specifically matched the framework version and k3s version of that so it was, in theory, a like-for-like build. The Dockerfile recently got a bump to the kairos-init version due to renovate bumping it, although I haven't tried that yet.

what config you used for k3s?

My /etc/90_custom.yaml and /etc/systemd/system/k3s-agent.service.d/override.conf files are in this gist: https://gist.github.com/sdwilsh/aef2ecc1579eb2de45c6edb902de0835

Is this a worker node in a cluster right?

Correct. I currently have three workers and a single control plane.

[sdwilsh]

@Itxaka, I also realized I experienced #3370 (which I just filed after recovering from a bumpy 3.4.x upgrade that went sideways) with the custom image, which might be another data point that the changes in builds between 3.3.x and 3.4.x could be at play here.

[Itxaka]

I will try this with a more similar environment, but for a single node, the upgrde from 3.3.1 to 3.4.0 worked as expected, with all pods being brought up with no problems, so maybe this affects workers and not masters?

[Itxaka]

umm, I tried with a master node on kairos 3.4.0 (k3s 1.31.7) and a worker with 3.3.1 (k3s 1.32.1)
Linked them and it was ok.

Then I upgraded the worker to 3.4.0 (k3s 1.32.3) and it came back with no issues...

kairos@worker1:~$ cat /etc/kairos-release 
KAIROS_ARCH="amd64"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_FAMILY="debian"
KAIROS_FIPS="false"
KAIROS_FLAVOR="ubuntu"
KAIROS_FLAVOR_RELEASE="24.04"
KAIROS_FRAMEWORK_VERSION="v2.21.0"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID="kairos"
KAIROS_ID_LIKE="kairos-standard-ubuntu-24.04"
KAIROS_IMAGE_LABEL="24.04-standard-amd64-generic-v3.4.0"
KAIROS_MODEL="generic"
KAIROS_NAME="kairos-standard-ubuntu-24.04"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_RELEASE="v3.4.0"
KAIROS_SOFTWARE_VERSION="1.32.3+k3s1"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_TARGETARCH="amd64"
KAIROS_VARIANT="standard"
KAIROS_VERSION="v3.4.0"
kairos@worker1:~$ systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/k3s-agent.service.d
             └─override.conf
     Active: active (running) since Mon 2025-04-28 09:56:07 UTC; 1min 48s ago
       Docs: https://k3s.io
   Main PID: 1755 (k3s-agent)
      Tasks: 38
     Memory: 445.4M (peak: 446.3M)
        CPU: 2.035s
     CGroup: /system.slice/k3s-agent.service
             ├─1755 "/usr/bin/k3s agent"
             ├─1923 "containerd "
             └─2423 /var/lib/rancher/k3s/data/9c5025828cb319b4b3aed5bd59ed36df271b889c6ce6206e39e8a0f75>

root@localhost:~# kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS        AGE
kube-system   coredns-ff8999cc5-bh5c7                   1/1     Running     1 (47m ago)     119m
kube-system   helm-install-traefik-crd-94xgm            0/1     Completed   0               47m
kube-system   helm-install-traefik-dj6mq                0/1     Completed   0               47m
kube-system   local-path-provisioner-5b5f758bcf-smlxs   1/1     Running     0               47m
kube-system   metrics-server-7bf7d58749-nxmnq           1/1     Running     0               47m
kube-system   svclb-traefik-2da1d792-4dv42              2/2     Running     2 (3m25s ago)   13m
kube-system   svclb-traefik-2da1d792-pbzf2              2/2     Running     0               47m
kube-system   traefik-5cbdcf97f4-lbjkb                  1/1     Running     0               47m
root@localhost:~# kubectl get nodes
NAME        STATUS   ROLES                  AGE    VERSION
localhost   Ready    control-plane,master   120m   v1.31.7+k3s1
worker1     Ready    <none>                 13m    v1.32.3+k3s1

The config for the workers is a very generic one:

#cloud-config

install:
    grub_options:
        extra_cmdline: rd.immucore.debug
    poweroff: false
    reboot: false
k3s-agent:
    args:
        - --node-name=worker1
    enabled: true
    env:
        K3S_TOKEN: K100449e8b11b0776f58432b767c2bf0285b8a4e1b6abcf7485bf0606eea2392996::7r7zsq.q95kgobagucsy3eb
        K3S_URL: https://192.168.122.50:6443
stages:
    initramfs:
        - name: Set user and password
          users:
            kairos:
                groups:
                    - admin
                passwd: kairos
        - hostname: worker1
          name: Set hostname
```

I did not set any bonds or flannel things.

@sdwilsh can you attach the k3s agent logs to see whats failing? Maybe in you case it needs some packages or something that are not on the default base install?