Untrusted slang shaders

[Ollhax]

Hi there, I’m working on a game project where I want to support modding as a first-class feature. Mods should be easy to share and safe to run. For regular CPU code I’ve set it up like chromium tabs, i.e. there’s a trusted engine process and a separate low-privilege mod process that communicates with the engine process through a controlled channel. I’m considering whether to support custom shaders in mods - what would I need to run them safely?

So say we have a malicious modder that wants to do harm through a .slang file, what could they do? What I can think of is the following:

1. Includes and imports reading sensitive files
For example, #include “C:/windows/...”. Write the output to a texture and read it from the mod. Share it to all players in the multiplayer session. Not good!

I could mitigate this by parsing includes and validating file paths, but I'm a bit worried it could be obfuscated with the preprocessor. I think I could use “-output-includes” to validate the resolved file paths. Is there any way to lock this down even more?

2. Cause GPU stalls / crashes
So there's multiple levels of problems here. Crashing the mod is fine, you can do that any time with CPU code. If they can make the engine process crash (the one running OpenGL/window management etc) with bad shader code that would be worse, but perhaps acceptable if it's not trivial to achieve.

Worse, perhaps someone crafty could exploit a GPU driver bug to make the system crash? Or worst possible case, there might be arbitrary code execution bugs. I hope that this is something driver vendors are keen on fixing these days with WebGPU etc.

I could also see the program allocating too much memory, to the point where the system is affected. But I could probably monitor for this and handle it.

3. Mess up the shader compiler
Feed in something that makes the shader compiler crash or behave badly. Perhaps the slang compiler has bugs that could be exploited? Maybe there are #pragmas or something that could trip it up? Extensions?

Having both the slang compiler and then opengl’s shader compilation means the attack surface is pretty big.

4. Compute shaders
Perhaps exposing compute shaders would be more problematic than vertex/pixel shaders? I haven’t done any research on this yet.

---

I’m not sure I want to go down this road since there’s a lot of uncertainty, but I’m hoping that it’s not as bad as running untrusted CPU code. I'm interesting to hear some thoughts on this - is there anything big and obvious I’m missing here?

If it’s relevant, I’m only supporting OpenGL and GLSL shaders right now.

[juliusikkala]

Stalls/deadlocks are easy to cause with shaders and can easily create big problems for the system and your game. Some drivers can freeze your PC permanently until reboot (AMD used to, although I haven't had this issue for a while), and others (Nvidia) freeze for some number of seconds until a watchdog kills the shader and your process loses its GPU session along with that. I'm not sure how webgpu implementations deal with that, but hopefully someone else can comment on that.

Currently, new ways to crash the Slang compiler are found on a weekly basis, so I would definitely not rely on it being exploit-free. I suspect a pro would probably find a successful exploit within days. It should definitely be sandboxed if you intend to compile untrusted on code on user's PCs. This could also help with your include concern - limit access only to files inside the sandbox.

Compute shaders themselves aren't more dangerous than fragment / vertex shaders, as the same "bad" things can be done in all shader types. They can all deadlock and try to dereference invalid pointers.

This subject is actually something I've thought of myself before as well, although it was for CPU code. I went as far as to implement an IR for it, from which bounded time and memory usage could be statically proven, but never did a high-level language on top of it. It'd be interesting to have a subset of SPIR-V for which we can guarantee a bounded maximum running time (allowing a bounded number of back-jumps) and safe memory accesses. Not sure if this already exists. That SPIR-V could then be distributed in the mod.

[Ollhax]

Thanks for your answer. I agree, sandboxing the slang compiler would probably be the best option if I go ahead with this.

[Devon7925]

You might want to try using the wasm build of the compiler, so you don't have to do full sandboxing.