mount propagation error on gVisor NFS plugin job

[abdulhp]

Hello,

I'm having trouble on deploying nfs csi plugin job on gVisor runtime. The error i encounter is

gvisor: reading spec: root mount propagation option must specify private or slave: \"shared\": OCI runtime error

How to specify the mount propagation option in csi_plugin stanza of NFS plugin job?

csi_plugin {
  id        = "nfs"
  type      = "controller"
  mount_dir = "/csi"
}

Additional Info:

Nomad v1.10.3
BuildDate 2025-07-08T18:46:27Z
Revision fde0bf26a7db35fc63ceb50b5b694478034392c6

podman version 4.6.2

runsc version release-20250721.0
spec: 1.2.0

[tgross]

Hi @abdulhp! The csi_plugin block only creates directories and a socket file to be created within the allocation directory of the plugin task, it doesn't create any new bind-mounts. What makes you suspect that CSI is involved here? Are you able to run other jobs with this runtime? Can you share the jobspec and client logs from the startup of the allocation and task?

[abdulhp]

Hello @tgross, thanks for your reply. Actually, I'm following the jobspec written in thatsk/nfs-csi-nomad and running properly with docker. I used CSI because I need to persist data such as wordpress directories. The directories is stored in separate NFS server.

After several attempt of testing Podman using gVisor, I'm able to run sample jobspec that provided by nomad. I try to deploy NFS CSI plugin but encountered with that error. I've tried to run the image without nomad(only podman + gvisor) and it runs. Thus, I assume the problem occurred from nomad-csi-plugin job or the nomad-driver-podman and I'm having no clue.

Jobspec

job "nfs-csi-plugin-podman" {
  namespace = "production"
  node_pool   = "gvisor"
  type = "system"
  priority = 100
  
  group "controller" {
    task "plugin" {
      driver = "podman"

      config {
        image = "registry.k8s.io/sig-storage/nfsplugin:v4.7.0"

        args = [
          "--v=5",
          "--nodeid=${attr.unique.hostname}",
          "--endpoint=unix:///csi/csi.sock",
          "--drivername=nfs.csi.k8s.io",
          "--logtostderr"
        ]
      }

      csi_plugin {
        id        = "nfs"
        type      = "controller"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 32
        memory = 32
      }
    }
  }

  group "node" {
    task "plugin" {
      driver = "podman"

      config {
        image = "registry.k8s.io/sig-storage/nfsplugin:v4.7.0"

        args = [
          "--v=5",
          "--nodeid=${attr.unique.hostname}",
          "--endpoint=unix:///csi/csi.sock",
          "--drivername=nfs.csi.k8s.io",
          "--logtostderr"
        ]

        # node plugins must run as privileged jobs because they
        # mount disks to the host
        privileged = true
      }

      csi_plugin {
        id        = "nfs"
        type      = "node"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 32
        memory = 32
      }
    }
  }
}

Client logs when submitting the jobspec

Aug 04 11:53:14 nomad-arm-g1 systemd[1]: Starting Nomad...
Aug 04 11:53:14 nomad-arm-g1 nomad[3870]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
Aug 04 11:53:14 nomad-arm-g1 nomad[3870]: ==> Starting Nomad agent...
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]: ==> Nomad agent configuration:
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:        Advertise Addrs: HTTP: 172.31.37.57:4646
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:             Bind Addrs: HTTP: [0.0.0.0:4646]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:                 Client: true
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:              Log Level: DEBUG
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:                 Region: global (DC: service-farm)
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:                 Server: false
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:                Version: 1.10.3
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]: ==> Nomad agent started! Log data will stream in below:
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.680+0700 [DEBUG] agent.plugin_loader: starting plugin: plugin_dir=/var/lib/nomad/plugins path=/var/lib/nomad/plugins/nomad-driver-podman args=["/var/lib/nomad/plugins/nomad-driver-podman"]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.680+0700 [DEBUG] agent.plugin_loader: plugin started: plugin_dir=/var/lib/nomad/plugins path=/var/lib/nomad/plugins/nomad-driver-podman pid=3878
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.680+0700 [DEBUG] agent.plugin_loader: waiting for RPC address: plugin_dir=/var/lib/nomad/plugins plugin=/var/lib/nomad/plugins/nomad-driver-podman
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.698+0700 [DEBUG] agent.plugin_loader.nomad-driver-podman: plugin address: plugin_dir=/var/lib/nomad/plugins address=/tmp/plugin2368519683 network=unix timestamp="2025-08-04T11:53:14.698+0700"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.698+0700 [DEBUG] agent.plugin_loader: using plugin: plugin_dir=/var/lib/nomad/plugins version=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.702+0700 [DEBUG] agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/var/lib/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.703+0700 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/var/lib/nomad/plugins plugin=/var/lib/nomad/plugins/nomad-driver-podman id=3878
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.703+0700 [DEBUG] agent.plugin_loader: plugin exited: plugin_dir=/var/lib/nomad/plugins
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.704+0700 [DEBUG] agent.plugin_loader.exec.id_validator: user range configured: plugin_dir=/var/lib/nomad/plugins denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.704+0700 [DEBUG] agent.plugin_loader.exec.id_validator: group range configured: plugin_dir=/var/lib/nomad/plugins denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.705+0700 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/plugins
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.705+0700 [DEBUG] agent.plugin_loader.raw_exec.id_validator: user range configured: plugin_dir=/var/lib/nomad/plugins denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.705+0700 [DEBUG] agent.plugin_loader.raw_exec.id_validator: group range configured: plugin_dir=/var/lib/nomad/plugins denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.705+0700 [DEBUG] agent.plugin_loader: starting plugin: plugin_dir=/var/lib/nomad/plugins path=/var/lib/nomad/plugins/nomad-driver-podman args=["/var/lib/nomad/plugins/nomad-driver-podman"]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.706+0700 [DEBUG] agent.plugin_loader: plugin started: plugin_dir=/var/lib/nomad/plugins path=/var/lib/nomad/plugins/nomad-driver-podman pid=3885
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.706+0700 [DEBUG] agent.plugin_loader: waiting for RPC address: plugin_dir=/var/lib/nomad/plugins plugin=/var/lib/nomad/plugins/nomad-driver-podman
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.720+0700 [DEBUG] agent.plugin_loader: using plugin: plugin_dir=/var/lib/nomad/plugins version=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.721+0700 [DEBUG] agent.plugin_loader.nomad-driver-podman: plugin address: plugin_dir=/var/lib/nomad/plugins address=/tmp/plugin3655398751 network=unix timestamp="2025-08-04T11:53:14.720+0700"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.723+0700 [DEBUG] agent.plugin_loader.nomad-driver-podman: http baseurl: plugin_dir=/var/lib/nomad/plugins url=unix:///run/podman/podman.sock @module=podman timestamp="2025-08-04T11:53:14.722+0700"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.723+0700 [DEBUG] agent.plugin_loader.nomad-driver-podman: http baseurl: plugin_dir=/var/lib/nomad/plugins @module=podman url=unix:///run/podman/podman.sock timestamp="2025-08-04T11:53:14.723+0700"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.724+0700 [DEBUG] agent.plugin_loader.stdio: received EOF, stopping recv loop: plugin_dir=/var/lib/nomad/plugins err="rpc error: code = Unavailable desc = error reading from server: EOF"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent.plugin_loader: plugin process exited: plugin_dir=/var/lib/nomad/plugins plugin=/var/lib/nomad/plugins/nomad-driver-podman id=3885
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [DEBUG] agent.plugin_loader: plugin exited: plugin_dir=/var/lib/nomad/plugins
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent: detected plugin: name=podman type=driver plugin_version=0.6.3
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.725+0700 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.727+0700 [INFO]  client: using state directory: state_dir=/var/lib/nomad/client
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.727+0700 [INFO]  client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.727+0700 [INFO]  client: using dynamic ports: min=20000 max=32000 reserved=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.730+0700 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=["arch", "bridge", "cgroup", "cni", "consul", "cpu", "host", "host_volume_plugins", "landlock", "memory", "network", "nomad", "plugins_cni", "signal", "storage", "vault", "env_aws", "env_gce", "env_azure", "env_digitalocean"]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.736+0700 [DEBUG] client.fingerprint_mgr.cgroup: detected cgroups: version=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.736+0700 [DEBUG] client.fingerprint_mgr: detected CNI network: name=nomad
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.744+0700 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul initial_period=15s
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.cpu: detected CPU efficiency core speed: mhz=2500
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.cpu: detected CPU performance core speed: mhz=0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.cpu: detected CPU efficiency core count: cores=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.cpu: detected CPU performance core count: cores=0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.cpu: detected CPU core count: cores=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.host_volume_plugins: detected plugin built-in: plugin_id=mkdir version=0.0.1
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.745+0700 [DEBUG] client.fingerprint_mgr.host_volume_plugins: plugin dir does not exist: dir=/var/lib/nomad/host_volume_plugins
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.747+0700 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=ens5
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.747+0700 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/ens5/speed device=ens5
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.747+0700 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected and no speed specified by user, falling back to default speed: interface=ens5 mbits=1000
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.747+0700 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=ens5 IP=172.31.37.57
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.748+0700 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.748+0700 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed device=lo
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.748+0700 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=lo mbits=1000
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.749+0700 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=ens5
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.749+0700 [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/ens5/speed device=ens5
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.749+0700 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=ens5 mbits=1000
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.749+0700 [DEBUG] client.fingerprint_mgr.cni_plugins: unexpected non-executable in cni plugin directory: name=LICENSE
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.749+0700 [DEBUG] client.fingerprint_mgr.cni_plugins: unexpected non-executable in cni plugin directory: name=README.md
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.782+0700 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault initial_period=15s
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.790+0700 [DEBUG] client.fingerprint_mgr.env_aws: could not read attribute value: attribute=public-ipv4 error="operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.794+0700 [DEBUG] client.fingerprint_mgr.env_aws: could not read attribute value: attribute=public-hostname error="operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.797+0700 [DEBUG] client.fingerprint_mgr.env_aws: could not read attribute value: attribute=network/interfaces/macs/06:4f:66:5c:ca:77/ipv6s error="operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.798+0700 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type resp_code=404
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.799+0700 [DEBUG] client.fingerprint_mgr.env_azure: could not read value for attribute: attribute=compute/azEnvironment resp_code=404
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [DEBUG] client.fingerprint_mgr.env_digitalocean: could not read value for attribute: attribute=region resp_code=404
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=["arch", "bridge", "cni", "consul", "cpu", "host", "host_volume_plugins", "network", "nomad", "plugins_cni", "signal", "storage", "env_aws"]
Aug 04 11:53:15 nomad-arm-g1 systemd[1]: Started Nomad.
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [INFO]  client.proclib.cg2: initializing nomad cgroups: cores=0-1
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [DEBUG] client.proclib.cg2: top level partition root nomad.slice cgroup initialized
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [DEBUG] client.proclib.cg2: partition member nomad.slice/share cgroup initialized
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [DEBUG] client.proclib.cg2: partition member nomad.slice/reserve cgroup initialized
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [INFO]  client.plugin: starting plugin manager: plugin-type=csi
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.800+0700 [INFO]  client.plugin: starting plugin manager: plugin-type=driver
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [INFO]  client.plugin: starting plugin manager: plugin-type=device
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.device_mgr: exiting since there are no device plugins
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [INFO]  client.plugin: starting plugin manager: plugin-type=dynamic_host_volume
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr: starting plugin: driver=podman path=/var/lib/nomad/plugins/nomad-driver-podman args=["/var/lib/nomad/plugins/nomad-driver-podman"]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr.exec.id_validator: user range configured: driver=exec denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr.exec.id_validator: group range configured: driver=exec denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr.docker: using client connection initialized from environment: driver=docker
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=healthy description=Healthy
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=undetected description=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.801+0700 [ERROR] client.driver_mgr.docker: failed to list pause containers for recovery: driver=docker error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr.docker: could not connect to docker daemon: driver=docker endpoint=unix:///var/run/docker.sock error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=undetected description="Failed to connect to docker daemon"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=dynamic_host_volume
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=dynamic_host_volume
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr.raw_exec.id_validator: user range configured: driver=raw_exec denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr.raw_exec.id_validator: group range configured: driver=raw_exec denied range=""
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=healthy description=Healthy
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr: plugin started: driver=podman path=/var/lib/nomad/plugins/nomad-driver-podman pid=3976
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.802+0700 [DEBUG] client.driver_mgr: waiting for RPC address: driver=podman plugin=/var/lib/nomad/plugins/nomad-driver-podman
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.804+0700 [DEBUG] client.consul: bootstrap contacting Consul DCs: consul_dcs=["service-farm"]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.806+0700 [INFO]  client.consul: discovered following servers: servers=[172.31.9.102:4647, 172.31.32.209:4647, 172.31.29.153:4647]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.806+0700 [DEBUG] client.server_mgr: new server list: new_servers=[172.31.29.153:4647, 172.31.32.209:4647, 172.31.9.102:4647] old_servers=[]
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.822+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: plugin address: driver=podman address=/tmp/plugin1360846639 network=unix timestamp="2025-08-04T11:53:14.822+0700"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.822+0700 [DEBUG] client.driver_mgr: using plugin: driver=podman version=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:14.824+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: http baseurl: driver=podman @module=podman url=unix:///run/podman/podman.sock timestamp="2025-08-04T11:53:14.824+0700"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.220+0700 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=podman health=healthy description="All Podman sockets are responding."
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.220+0700 [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[exec raw_exec podman] undetected:[qemu java docker]]"
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.220+0700 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.220+0700 [INFO]  client: started client: node_id=f1a4efcc-2474-a862-e4b7-c62a988d5fab
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.220+0700 [WARN]  agent: not registering Nomad HTTPS Health Check because verify_https_client enabled
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.222+0700 [DEBUG] http: UI is enabled
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.223+0700 [DEBUG] http: UI is enabled
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.223+0700 [DEBUG] client: updated allocations: index=7053335 total=0 pulled=0 filtered=0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.224+0700 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.224+0700 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.241+0700 [INFO]  client: node registration complete
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.241+0700 [DEBUG] client: evaluations triggered by node registration: num_evals=2
Aug 04 11:53:15 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:15.242+0700 [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=0 deregistered_checks=0 failures=0
Aug 04 11:53:22 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:22.165+0700 [DEBUG] client: state changed, updating node and re-registering
Aug 04 11:53:22 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:22.177+0700 [INFO]  client: node registration complete
Aug 04 11:53:28 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:28.266+0700 [DEBUG] http: request complete: method=GET path=/v1/metrics?format=prometheus duration=2.707453ms
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.439+0700 [DEBUG] client: updated allocations: index=7053344 total=2 pulled=2 filtered=0
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.439+0700 [DEBUG] client: allocation updates: added=2 removed=0 updated=0 ignored=0
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.457+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin type=Received msg="Task received by client" failed=false
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.458+0700 [DEBUG] client.alloc_runner.widmgr: retrieving and renewing workload identities: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 num_identities=0
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.461+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin type=Received msg="Task received by client" failed=false
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.461+0700 [DEBUG] client: allocation updates applied: added=2 removed=0 updated=0 ignored=0 errors=0
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.462+0700 [DEBUG] client.alloc_runner.widmgr: retrieving and renewing workload identities: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 num_identities=0
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.469+0700 [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.469+0700 [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.479+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin type="Task Setup" msg="Building Task Directory" failed=false
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.481+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin type="Task Setup" msg="Building Task Directory" failed=false
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.493+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin path=/usr/bin/nomad args=["/usr/bin/nomad", "logmon"]
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.493+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin path=/usr/bin/nomad pid=4098
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.493+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin plugin=/usr/bin/nomad
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.495+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin path=/usr/bin/nomad args=["/usr/bin/nomad", "logmon"]
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.497+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin path=/usr/bin/nomad pid=4099
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.497+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin plugin=/usr/bin/nomad
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.540+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin @module=logmon address=/tmp/plugin1734846604 network=unix timestamp="2025-08-04T11:53:41.540+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.540+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin version=2
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.541+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin version=2
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.541+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin network=unix @module=logmon address=/tmp/plugin805722165 timestamp="2025-08-04T11:53:41.541+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.547+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin path=/var/lib/nomad/alloc/e0b39a2d-73eb-aa09-b709-3f4688b6f2b9/alloc/logs/.plugin.stdout.fifo @module=logmon timestamp="2025-08-04T11:53:41.547+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.547+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin @module=logmon path=/var/lib/nomad/alloc/bedcf8ea-2658-9b51-6004-4310bb29e718/alloc/logs/.plugin.stdout.fifo timestamp="2025-08-04T11:53:41.547+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.548+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin @module=logmon path=/var/lib/nomad/alloc/bedcf8ea-2658-9b51-6004-4310bb29e718/alloc/logs/.plugin.stderr.fifo timestamp="2025-08-04T11:53:41.547+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.548+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin @module=logmon path=/var/lib/nomad/alloc/e0b39a2d-73eb-aa09-b709-3f4688b6f2b9/alloc/logs/.plugin.stderr.fifo timestamp="2025-08-04T11:53:41.547+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.596+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: applied labels on the container: driver=podman @module=podman labels="map[com.hashicorp.nomad.alloc_id:e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 com.hashicorp.nomad.job_id:nfs-csi-plugin-podman com.hashicorp.nomad.job_name:nfs-csi-plugin-podman com.hashicorp.nomad.namespace:production com.hashicorp.nomad.node_id:f1a4efcc-2474-a862-e4b7-c62a988d5fab com.hashicorp.nomad.node_name:nomad-arm-g1 com.hashicorp.nomad.task_group_name:controller com.hashicorp.nomad.task_name:plugin]" timestamp="2025-08-04T11:53:41.596+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.596+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: applied labels on the container: driver=podman @module=podman labels="map[com.hashicorp.nomad.alloc_id:bedcf8ea-2658-9b51-6004-4310bb29e718 com.hashicorp.nomad.job_id:nfs-csi-plugin-podman com.hashicorp.nomad.job_name:nfs-csi-plugin-podman com.hashicorp.nomad.namespace:production com.hashicorp.nomad.node_id:f1a4efcc-2474-a862-e4b7-c62a988d5fab com.hashicorp.nomad.node_name:nomad-arm-g1 com.hashicorp.nomad.task_group_name:node com.hashicorp.nomad.task_name:plugin]" timestamp="2025-08-04T11:53:41.596+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.678+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: Found imageID: driver=podman @module=podman c8215a0d2abc7875409a020c36252d74cd0d87e1342ed293d4afa09aa603b3af="for image" registry.k8s.io/sig-storage/nfsplugin:v4.7.0="in local storage" timestamp="2025-08-04T11:53:41.678+0700"
Aug 04 11:53:41 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:41.678+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: Found imageID: driver=podman @module=podman c8215a0d2abc7875409a020c36252d74cd0d87e1342ed293d4afa09aa603b3af="for image" registry.k8s.io/sig-storage/nfsplugin:v4.7.0="in local storage" timestamp="2025-08-04T11:53:41.678+0700"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.169+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: Cleaning up: driver=podman @module=podman container=326e6fc45cc1d35d2ac22942776ec2f3a02195bce7a3fcb08e76cf90a4fdaf42 timestamp="2025-08-04T11:53:42.169+0700"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.174+0700 [DEBUG] client.driver_mgr.nomad-driver-podman: Cleaning up: driver=podman container=96cc251ec83a4512b84e27a338880e404619b7f1af1652107309f3839849859d @module=podman timestamp="2025-08-04T11:53:42.174+0700"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.249+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin type="Driver Failure"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   msg=
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   | rpc error: code = Unknown desc = failed to start task, could not start container: cannot start container, status code: 500: {"cause":"OCI runtime error","message":"gvisor: reading spec: root mount propagation option must specify private or slave: \"shared\": OCI runtime error","response":500}
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:    failed=false
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.252+0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   error=
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   | rpc error: code = Unknown desc = failed to start task, could not start container: cannot start container, status code: 500: {"cause":"OCI runtime error","message":"gvisor: reading spec: root mount propagation option must specify private or slave: \"shared\": OCI runtime error","response":500}
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.252+0700 [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin reason="Error was unrecoverable"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.252+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin type="Not Restarting" msg="Error was unrecoverable" failed=true
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.261+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin type="Driver Failure"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   msg=
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   | rpc error: code = Unknown desc = failed to start task, could not start container: cannot start container, status code: 500: {"cause":"OCI runtime error","message":"gvisor: reading spec: root mount propagation option must specify private or slave: \"shared\": OCI runtime error","response":500}
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:    failed=false
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.263+0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   error=
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   | rpc error: code = Unknown desc = failed to start task, could not start container: cannot start container, status code: 500: {"cause":"OCI runtime error","message":"gvisor: reading spec: root mount propagation option must specify private or slave: \"shared\": OCI runtime error","response":500}
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:   
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.263+0700 [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin reason="Error was unrecoverable"
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.263+0700 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin type="Not Restarting" msg="Error was unrecoverable" failed=true
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.352+0700 [DEBUG] client: updated allocations: index=7053346 total=2 pulled=0 filtered=2
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.352+0700 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2
Aug 04 11:53:42 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:42.352+0700 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0
Aug 04 11:53:43 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:43.258+0700 [DEBUG] http: request complete: method=GET path=/v1/metrics?format=prometheus duration=2.94173ms
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.260+0700 [WARN]  client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin @module=logmon timestamp="2025-08-04T11:53:46.260+0700"
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.262+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.stdio: received EOF, stopping recv loop: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin err="rpc error: code = Unavailable desc = error reading from server: EOF"
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]: ==> Caught signal: broken pipe
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.265+0700 [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin plugin=/usr/bin/nomad id=4098
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.265+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.265+0700 [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 task=plugin
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.266+0700 [INFO]  client.gc: marking allocation for GC: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.271+0700 [WARN]  client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin @module=logmon timestamp="2025-08-04T11:53:46.271+0700"
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.273+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.stdio: received EOF, stopping recv loop: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin err="rpc error: code = Unavailable desc = error reading from server: EOF"
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]: ==> Caught signal: broken pipe
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.275+0700 [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin plugin=/usr/bin/nomad id=4099
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.275+0700 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.276+0700 [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 task=plugin
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.276+0700 [INFO]  client.gc: marking allocation for GC: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.439+0700 [DEBUG] client: updated allocations: index=7053347 total=2 pulled=0 filtered=2
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.439+0700 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=2
Aug 04 11:53:46 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:46.439+0700 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=2 errors=0
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.744+0700 [DEBUG] client: updated allocations: index=7053351 total=0 pulled=0 filtered=0
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.744+0700 [DEBUG] client: allocation updates: added=0 removed=2 updated=0 ignored=0
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.744+0700 [DEBUG] client: allocation updates applied: added=0 removed=2 updated=0 ignored=0 errors=0
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.744+0700 [INFO]  client.gc: garbage collecting allocation: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9 reason="forced collection"
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.744+0700 [INFO]  client.gc: garbage collecting allocation: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718 reason="forced collection"
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.752+0700 [DEBUG] client.gc: alloc garbage collected: alloc_id=e0b39a2d-73eb-aa09-b709-3f4688b6f2b9
Aug 04 11:53:56 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:56.754+0700 [DEBUG] client.gc: alloc garbage collected: alloc_id=bedcf8ea-2658-9b51-6004-4310bb29e718
Aug 04 11:53:57 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:57.536+0700 [DEBUG] client: updated allocations: index=7053353 total=0 pulled=0 filtered=0
Aug 04 11:53:57 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:57.536+0700 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
Aug 04 11:53:57 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:57.536+0700 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
Aug 04 11:53:58 nomad-arm-g1 nomad[3870]:     2025-08-04T11:53:58.258+0700 [DEBUG] http: request complete: method=GET path=/v1/metrics?format=prometheus duration=2.831804ms

[tgross]

Ok, thanks for that output. What I can see from the logs is that you end up with two allocations (one per group), each with one task. Both tasks fail the the same error you've described here.

You said:

After several attempt of testing Podman using gVisor, I'm able to run sample jobspec that provided by nomad.

Are you referring to the example jobspec from nomad job init or this CSI job, which as you've noted doesn't seem to work?

Have you been able to run any Nomad job with podman and gVisor? The reason I ask is because during task setup Nomad isn't doing anything special for CSI jobs here. So I'd expect the bind-mounted allocation directory we pass to fail for all jobs and not just CSI jobs.

Also, are sure that gVisor works with CSI at all? I found google/gvisor#9324 and I'm realizing that the sandbox may make it challenging for CSI plugins to perform disk operations on the host.

[abdulhp]

Hmm, interesting @tgross. I've read the google/gvisor#9324. I felt unsure at first.

By the way, i used example jobspec that provided in nomad UI. I clicked Run Job button and then clicked Choose a template.

Also, i've tried to run redis with bind-mounted allocation directory. I need to modify the permission and ownership of the mounted directory because i kept getting permission denied (2025-08-11T16:14:59.259326005+07:00 stdout F 1:M 11 Aug 2025 09:14:59.259 # Can't open or create append-only dir appendonlydir: Permission denied).

And... the job run perfectly, i tried to do rdb snapshot and the rdb file stored successfully.

Here's the detail.
nomad.hcl

...
client {
  ...
  
  host_volume "redis-dir" {
    path      = "/home/ubuntu/redis-dir"
    read_only = false
  }
}
...

Redis Job

job "redis-bind-mount" {
  type = "service"
  node_pool = "gvisor"

  group "redis" {
    count = 1

    network {
      port "db" {
        static = 6379
      }
    }

    volume "redis-dir" {
      type = "host"
      source = "redis-dir"
    }

    task "fix-permissions" {
      driver = "podman"
      
      config {
        image = "alpine:latest"
        command = "sh"
        args = [
          "-c",
          "mkdir -p /redis/data && chown -R 1001:1001 /redis/data && chmod -R 755 /redis/data"
        ]
      }

      volume_mount {
        volume      = "redis-dir"
        destination = "/redis"
        read_only   = false
      }

      lifecycle {
        hook    = "prestart"
        sidecar = false
      }
    }

    task "redis" {
      driver = "podman"

      config {
        image = "bitnami/redis:latest"
        args = [
          "redis-server",
          "/redis/redis.conf"
        ]
      }

      env {
        ALLOW_EMPTY_PASSWORD = "yes"
        REDIS_DISABLE_COMMANDS = ""
      }

      volume_mount {
        volume      = "redis-dir"
        destination = "/redis"
        read_only   = false
      }

      action "backup" {
        command = "redis-cli"
        args    = ["-h", "127.0.0.1", "-p", "6379", "BGSAVE"]
      }

      action "save-backup" {
        command = "sh"
        args = [
          "-c", 
          "redis-cli -h 127.0.0.1 -p 6379 BGSAVE && sleep 5 && cp /redis/data/dump.rdb /redis/data/dump-$(date +%Y%m%d-%H%M%S).rdb"
        ]
      }
    }
  }
}

[tgross]

Ok, just so that I understand what you're saying here:

• 1. You ran the redis example job from the web UI with driver = "podman" but otherwise unaltered, and it worked?
• 2. You also ran the jobspec you've shown above with a volume block and that didn't work? (Permission denied errors)

Are both (1) and (2) correct?

[abdulhp]

Exactly @tgross

[tgross]

Ok, thanks. I went back and looked at how we implement the CSI plugin and realized that we do in fact have mounts we're configuring in the plugin supervisor hook (ref plugin_supervisor_hook.go#L167-L180), so I was wrong in my earlier comment

But that also tells me why CSI plugins must mount these in bidirectional mode: the plugin needs to be able to make mount changes that are reflected on the host, because that's what CSI does! So I don't really see any path to make CSI compatible with gVisor at all. I don't think this is really a problem specific to Nomad (and certainly not to the podman driver).