Podman on WSL2 fails kind create cluster as root and not as root

[erharb]

What happened:
Attempting to replace Docker Desktop for Windows on WSL2 with Podman leads to KIND failures that I am unable to figure out how to work around at the moment.

Running KIND with a non-root account leads to the following errors:
wsl2_podman_kind.log

notroot@wsl2:~$ kind -v5 create cluster
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
ERROR: failed to create cluster: running kind with rootless provider requires cgroup v2, see https://kind.sigs.k8s.io/docs/user/rootless/

Running KIND as root leads to different error and times out much later on, but seems to prove that it doesn't appear to be an actual problem with cgroup v1/v2.

root@wsl2:~# /root/go/bin/kind -v5 create cluster
enabling experimental podman provider
Creating cluster "kind" ...
DEBUG: podman/images.go:58] Image: kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼
 ✓ Preparing nodes 📦
...
 ✓ Writing configuration 📜
DEBUG: kubeadminit/init.go:81] I0907 02:05:08.207219     198 initconfiguration.go:246] loading configuration from "/kind/kubeadm.conf"
...
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
...
 ✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "podman exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
...
couldn't initialize a Kubernetes cluster

What you expected to happen:
Kind to create a cluster successfully on WSL using Podman

How to reproduce it (as minimally and precisely as possible):
Start with fresh Ubuntu 20.04 WSL2 image:

1. Installed latest version of Podman
2. Installed latest version of Go 1.15.x
3. Installed Kind GO111MODULE="on" go get sigs.k8s.io/kind@v0.11.1

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.11.1 go1.15.15 linux/amd64
• Kubernetes version: (use kubectl version): not installed
• Podman version: (use podman info): 3.2.3
• OS (e.g. from /etc/os-release): Ubuntu 20.04.3 LTS (kernel: 5.10.16.3-microsoft-standard-WSL2)

[JackPott]

I just attempted the same thing and can see the same problem.

sudo kind create cluster --retain
sudo kind export logs

Looking at logs/kind-control-plane/kubelet.log there may be some clues

Sep 07 14:58:03 kind-control-plane systemd[1]: Condition check resulted in kubelet: The Kubernetes Node Agent being skipped.
Sep 07 14:58:09 kind-control-plane systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Sep 07 14:58:09 kind-control-plane systemd[1]: Started kubelet: The Kubernetes Node Agent.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: I0907 14:58:09.184673     232 server.go:197] "Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead"
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 14:58:09 kind-control-plane kubelet[232]: I0907 14:58:09.407335     232 server.go:440] "Kubelet version" kubeletVersion="v1.21.1"
Sep 07 14:58:09 kind-control-plane kubelet[232]: I0907 14:58:09.407552     232 server.go:851] "Client rotation is on, will bootstrap in background"
Sep 07 14:58:09 kind-control-plane kubelet[232]: I0907 14:58:09.410538     232 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Sep 07 14:58:09 kind-control-plane kubelet[232]: I0907 14:58:09.410578     232 container_manager_linux.go:995] "CPUAccounting not enabled for process" pid=232
Sep 07 14:58:09 kind-control-plane kubelet[232]: I0907 14:58:09.410588     232 container_manager_linux.go:998] "MemoryAccounting not enabled for process" pid=232
Sep 07 14:58:09 kind-control-plane kubelet[232]: E0907 14:58:09.419686     232 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post "https://kind-control-plane:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp [fc00:f853:ccd:e793::5]:6443: connect: connection refused
Sep 07 14:58:11 kind-control-plane kubelet[232]: E0907 14:58:11.498367     232 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post "https://kind-control-plane:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp [fc00:f853:ccd:e793::5]:6443: connect: connection refused
Sep 07 14:58:14 kind-control-plane kubelet[232]: W0907 14:58:14.417070     232 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Sep 07 14:58:14 kind-control-plane kubelet[232]: E0907 14:58:14.422375     232 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: invalid configuration: cgroup-root [\"kubelet\"] doesn't exist"
Sep 07 14:58:14 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 07 14:58:14 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 07 14:58:15 kind-control-plane systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Sep 07 14:58:15 kind-control-plane systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Sep 07 14:58:15 kind-control-plane systemd[1]: Starting kubelet: The Kubernetes Node Agent...

[BenTheElder]

There have been some recent issues with WSL2 + kind and the custom WSL2 init that should be patched soon and would apply to docker or podman #2390, the last comment in this thread appears to be that issue.

FWIW though I would recommend trying docker in a WSL2 VM w/o docker desktop.
Podman support has been more brittle and is still marked experimental because of those problems. It is a less stable development target.

[hwdef]

Having the same problem, is there any solution so far?

[mwoodpatrick]

Has there been any progress on getting podman to work with kind?

[BenTheElder]

Has there been any progress on getting podman to work with kind?

It generally works, it's unclear what this specific issue is. We've had some compatibility issues with podman changes in the past but as far as I know all current issues are fixed.

I still recommend docker for now, it's just a more stable target to integrate against and kind can support things like restarts properly.