podman in production: unpriviliged user vs. --userns=auto

[tobwen]

Based on this discussion, @rhatdan stated that podman run --userns=auto might be a better solution for containers in production launched by podman. For sure, when starting it as a privileged user, you're way more flexible.

But is the isolation in a unique UserNS as secure as running podman from an unprivileged user?

[rhatdan]

It is actually more secure. If you run two containers as a rootless user, they run in the same user namespace so they can attack each other from a User Namespace point of view.

If you run two containers as root with podman run --userns auto, then they run in unigue user namespace and are isolated.

Rootless containers are great for containers run by users on a system, but if you are just running containers on a server, then --userns=auto is a more secure solution. (I plan on writing a blog on this).

[sbrivio-rh]

Don't put all of your trust into SELinux when you're outside of RedHat. Seems like many Linux distributions have issues with it.
Article: https://klecko.github.io/posts/selinux-bypasses/

...which explains how to disable SELinux on some Android devices. 😕

@sbrivio-rh: in a rootfull podman networking context, what is the worst that malicious packets can do if SELinux is enabled and correctly configured?

Exactly the same as malicious packets can do without Podman, or without SELinux. You can spoof ARP, craft semi-random things to crash/DoS routers, etc. Network isolation is still there at Layer-2, but all you need to do is to get control over the network interface in the container to send out whatever you want. I covered that quickly in slide 6 of this deck for this presentation.

[Luap99]

But doesn't spoofing and crafting require CAP_NET_RAW? This is not given to a container by default and I see very little reason to do so and if an application really needs this they most likely cannot use pasta/slirp4netns anyway unless they only use it for local connections inside the netns.

[tobwen]

...which explains how to disable SELinux on some Android devices. 😕

I knew the comment was coming, so I copied the discussion page 😃. Some users say that SELinux unfortunately isn't always implemented reliably.

[sbrivio-rh]

But doesn't spoofing and crafting require CAP_NET_RAW? This is not given to a container by default

Sure, but if you start the container as root, you rely on Podman to drop it (and here I'm assuming a bug in Podman). Or one could also pass --privileged for whatever other reason and miss this aspect.

On top of that, what is sent to the container namespace (not necessarily to processes running into it) is not as restricted.

I knew the comment was coming, so I copied the discussion page 😃. Some users say that SELinux unfortunately isn't always implemented reliably.

Yeah, I read that as well, but... it starts with a mention of SELinux on Debian as an experiment. Now, sure, some people spent effort to get that somehow working, but the "native" LSM on Debian is AppArmor. It's not so much of a matter of SELinux implementation, it's rather about choices, configuration and integration.

[rhatdan]

SELinux by default implements unconfined networking, we rely on network namespace and dropped capabilities to protect the network.

If you start with the assumption that Podman is broken then running containers as root is a risk. Same as with the Kernel is broken or systemd or ...

SELinux confines activity within the container, especially about the file system, and using kernel file systems.

[tobwen]

Can you please remember to write a short blog post stating that a rootful podman is safer than a rootless podman? I'm really struggling to convince other users of this. Maybe a short notice in the README would be enough?

[Luap99]

Well this sentence is not true. We explicitly talk about --userns auto is more secure. This is a very important difference.
The secure part is that with userns auto all containers run in a different user namespace (different uids on the host). You can achieve the same as rootless user so I wouldn't say rootful is more secure than rootless.

[tobwen]

Sure, --userns auto is mandatory. However, I assumed that there are other security functions in the kernel that could currently only be accessed by root. It is wonderful that this has now been clarified.

[rhatdan]

I have a blog in progress, the problem is it is arguable which is more secure.
$ sudo podman run --userns=auto ... Runs each containers in a separate user namespace where each container is running with different UIDs and almost assuradly does not match any other UIDs on the host. BUT the podman command runs as root which means it pulls images as root. Pulling images as root can be risky, because flaws in the pulling could (have in the past) cause root to write anywhere.
$ podman run ... Runs the container in a single user namespace where it could attack the users homedir and potentially get a users data. Also all containers run in the same user namespace.

[tobwen]

$ podman run ... Runs the container in a single user namespace where it could attack the users homedir and potentially get a users data. Also all containers run in the same user namespace.

Would this also be the case if you run each container under a different user? Or would a different namespace then be assigned?

[rhatdan]

If you run each container under a different user and managed the /etc/subuid and /etc/subgid files to maintain separation, this would be safer, although they could access content in user homedir.

$ podman run --userns=nomap ...

Would solve this problem.

But this is way more complex. You could also setup a Huge /etc/subuid range for a user and then run lots of containers for that user with --userns=auto ... Which is probably the most secure.

[cgwalters]

One thing related to this...what would I think be quite cool is integration with systemd DynamicUser=yes at least for quadlet containers (we'd still need to mount the image as privileged, so doing this would probably require some lower level systemd integration).

[cgwalters]

Yeah, the containers user here is IMO pretty suboptimal. Is something actually intended to create that user today automatically?

[pomology]

@rhatdan, were you ever able to write the blog post above regarding the security implications of rootfull podman as potentially more secure vs rootless? I'd love to read it if you can share a link. Thank you!

[rhatdan]

@pomology Send me and email and I can probably preview what I have now to you.

[pomology]

@rhatdan Thank you! I sent you an email just now.

[tobwen]

I'm also interested. Do a MEAP, I'm a legit owner of https://www.manning.com/books/podman-in-action

[rhatdan]

Send me an email, and I will send the preliminary doc to you.

[kavishgr]

I'm also interested. I sent you an email.

[rdbisme]

Hello, what's the best practice here nowadays?

I'm currently deploying containerized services creating a dedicated non-root user on my system per "pod". Enable lingering and running the container within that dedicated user in rootless mode.

How does it look? Would it be better to run them instead with root + --userns=auto?

[giuseppe]

from a security PoV at runtime they behave in the same way. The main difference of using --userns=auto compared to a different system user per pod is that the code for pulling the image and setting up the container is also running in a user namespace. So the "user per pod" setup is slightly safer, than running as root with --userns=auto`. The cost is also higher though, since you are not able to share storage among different users

[videoMonkey]

I was very excited at the thought of not creating a user per pod but I am running into a problem if I run sudo podman run -d --name vaultwarden --userns=auto -p 8000:80 docker.io/vaultwarden/server:latest I get the following feedback

ERRO[0000] Cannot find mappings for user "containers": no subuid ranges found for user "containers" in /etc/subuid
Error: creating container storage: not enough unused IDs in user namespace

and maybe I need to read more about how this all works, but I tried setting subuid's and subgid's for the user 'container', that did nothing, I tried doing it for root and then realized root already had a lot. I tried running the command without sudo, that worked.

I was hoping to have an uncomplicated and secure way to have multiple admins be able to tinker with containers and this seemed to be that promise. If i could get it going....

[sbrivio-rh]

Could that be https://bugzilla.redhat.com/show_bug.cgi?id=2382662?

[vmsh0]

ERRO[0000] Cannot find mappings for user "containers": no subuid ranges found for user "containers" in /etc/subuid
Error: creating container storage: not enough unused IDs in user namespace

What you're saying is correct ("I was hoping to have an uncomplicated and secure way to have multiple admins be able to tinker with containers and this seemed to be that promise.").

But this functionality requires some initial setup in the operating system for the users running the containers. You should read subuid(5) and set up proper mappings, as reported by the error message. This can also be easily integrated in user creation (and it is, in some distros.)

[vmsh0]

I.e., "auto" does not mean that the system configures itself automatically. "auto" means that a suitable ID range is selected from the available range.

[jzcad1828]

I think I also ran across this issue. I added the example from this documentation to my gid files and it fixed it for me:
https://docs.podman.io/en/v4.6.1/markdown/options/userns.container.html:

The --userns=auto flag requires that the user name containers be specified in the /etc/subuid and /etc/subgid files, with an unused range of subordinate user IDs that Podman containers are allowed to allocate. See subuid(5).

Example: containers:2147483647:2147483648

[pierrecnalb]

I am a bit surprised by the conclusion of this discussion:
why is the recommendation for servers to use rootful containers with --userns=auto, instead of using a dedicated rootless user with --userns=auto?
Basically, instead of adding the containers:2147483647:2147483648 to /etc/subuid and /etc/subgid, why not adding a real dedicated user that would run all containers in rootless mode?
Something like:

$ cat /etc/subuid
dwalsh:100000:65536
rootless:2147483647:2147483648
$ cat /etc/subgid
dwalsh:100000:65536
rootless:2147483647:2147483648

where rootless would be a real user and all containers and pods would be running with that user using --userns=auto

Any comment on that approach @rhatdan, @giuseppe ?

[pierrecnalb]

Many thanks for the clarification!
By "easier to manage", is that only because root already exist as a user? or is that due to some complex differences with how rootless works (e.g. the pasta network stack etc?)?
In other words, are there any known big difficulties that would prevent it from being the recommended approach for production servers?

[giuseppe]

you need to make sure the user session is always kept alive (you can set lingering for that user), otherwise containers are also killed when the user session is terminated. With root you don't have this problem

[pierrecnalb]

Got it! Thanks!
Could you confirm though, that for rootless, it is enough to increase the range of uid/guid for the dedicated user in /etc/subuid and /etc/subgid and the --userns=auto option would work similarly as root with its containers entry?
That is, instead of the following for rootful containers,

$ cat /etc/subuid
me:100000:65536
containers:2147483647:2147483648
$ cat /etc/subgid
me:100000:65536
containers:2147483647:2147483648

I would have the following for rootless containers running with rootless user:

$ cat /etc/subuid
me:100000:65536
rootless:2147483647:2147483648
$ cat /etc/subgid
me:100000:65536
rootless:2147483647:2147483648

Or is there anything else to be aware of?

[giuseppe]

yes, that is enough.

Of course the usual rootless limitations apply, especially when it comes to networks

[sbrivio-rh]

Beware of #17840 as well. But as far as I understand there's a workaround in place for the moment that should make things work.

[videoMonkey]

My struggle was with the advice pointed to in this thread. To make a user and attach a container to it this is what I've done, vaultwarden is an example of what I am trying to do. This has worked just fine for me. `sudo useradd vaultwarden` `sudo loginctl enable-linger vaultwarden` `mkdir -p /home/vaultwarden/vaultwarden_data` `mkdir -p /home/vaultwarden/.config/containers/systemd` `sudo chown -R vaultwarden:vaultwarden /home/vaultwarden/` to make sure the following command works `sudo dnf install systemd-container` `sudo machinectl shell vaultwarden@` or potentially `sudo -i -u vaultwarden` I'm not sure what the ups and downs are ``` cat < ~/.config/containers/systemd/vaultwarden.container [Unit] Description=Vaultwarden Server [Container] EnvironmentFile=%h/.env Image=docker.io/vaultwarden/server:latest AutoUpdate=registry Network=host UserNS=keep-id Volume=%h/vaultwarden_data:/data:Z Environment=ROCKET_PORT=8000 [Service] # Inform systemd of additional exit status Success ExitStatus=0 143 # Extend Timeout to allow time to pull the image TimeoutStartSec=900 [Install] # Start by default on boot WantedBy=default.target # https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html EOF ``` `systemctl --user daemon-reload` `systemctl --user start vaultwarden.service` `systemctl --user status vaultwarden.service` This has worked pretty well. But I would like to dedicate a moment to try the tehnique mentioned in this issue, I think it would make management easier. Message ID: ***@***.***>