containerd in MicroK8s does not pass `runtime.options` TOML to custom shim

[kate-goldenring]

Description:

I'm running a custom containerd shim on MicroK8s and encountering a bug where the [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.<name>.options] TOML configuration does not appear to be passed to the shim at runtime.

This works as expected in Minikube, but fails in MicroK8s with the same containerd configuration. This pull request has more details along with examples of how CI succeeds on all runtimes except MicroK8s: spinframework/runtime-class-manager#387 (comment)

---

Repro Steps:

1. Create a MicroK8s cluster with the following in /var/snap/microk8s/current/args/containerd-template.toml:

   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin-v2]
   runtime_type = "/opt/rcm/bin/containerd-shim-spin-v2"
   
   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin-v2.options]
   systemd_cgroup = true

2. Restart containerd (systemctl restart snap.microk8s.daemon-containerd) and attempt to run a pod with runtimeClassName: spin-v2.

3. Observe in the custom shim logs that config_body is empty or missing.

   Error:

    Warning  FailedCreatePodSandBox  0s (x10 over 9s)  kubelet            Failed to create pod sandbox: rpc error: 
    code = InvalidArgument desc = failed to create containerd task: failed to create shim task: invalid shim options: 
    invalid argument: invalid shim options with body : TOML parse error at line 1, column 1
      |
    1 | 
      | ^
    missing field `systemd_cgroup`
    : invalid argument

4. When running the same shim and config on Minikube, the TOML in options is correctly passed via the config_body in the CreateTaskRequest.

---

Expected Behavior:

Containerd should pass the contents of the options block as TOML in config_body in the CreateTaskRequest.Options.

---

Actual Behavior:

config_body is either empty or missing entirely in MicroK8s, even though the runtime config appears valid and works on other distributions.

---

Environment:

• MicroK8s version: v1.32.3 revision 8148
• Host OS: Ubuntu 22.04
• Custom shim: containerd-shim-spin-v2: https://github.com/spinframework/containerd-shim-spin

---

The same configuration and shim run fine in Minikube and other containerd environments.

---

Suggested Fix / Workaround:

• Confirm whether MicroK8s is writing the options block into containerd's live config
• Consider verifying behavior in containerd config dump
• Ensure that custom runtime.options blocks are not stripped or ignored when containerd is launched via snap

---

Let me know if you'd like me to provide a minimal reproduction with a dummy shim binary.

---

Would you like me to submit the issue for you or draft a separate reproduction repo?

[berkayoz]

Hey @kate-goldenring,

I believe we are not performing anything specific related to these options. The template is ultimately rendered to /var/snap/microk8s/current/args/containerd.toml on containerd startup. Is there a specific containerd version we expect?

I've added the given section on the containerd-template.toml, performed a restart. Here's the containerd config dump:

containerd config dump

root@microk8s:~# /snap/microk8s/current/bin/containerd --config /var/snap/microk8s/current/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock config dump
disabled_plugins = []
imports = ["/var/snap/microk8s/current/args/containerd.toml"]
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    drain_exec_sync_io_timeout = "0s"
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_deprecation_warnings = []
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.k8s.io/pause:3.10"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/var/snap/microk8s/8148/opt/cni/bin"
      conf_dir = "/var/snap/microk8s/8148/args/cni-network"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.kata.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
            BinaryName = "kata-runtime"

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime.options]
            BinaryName = "nvidia-container-runtime"

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin-v2]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "/opt/rcm/bin/containerd-shim-spin-v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin-v2.options]
            systemd_cgroup = true

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/var/snap/microk8s/8148/args/certs.d"

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.sideload-images"]
    namespace = ""
    sources = []

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    mount_options = []
    root_path = ""
    sync_remove = false
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

Maybe we could cross-reference from other deployments to see if we're missing a section. To me it seems like we're atleast giving containerd the right options.

[kate-goldenring]

I agree that the fields are properly loaded from the containerd-template.toml to containerd.toml. The strange part is that containerd is not passing the runtime options to the shim runtime.