Devenv-produced container startup broken between 1.8.0 and 1.8.1

[mcdonc]

Describe the bug

I produce a container under devenv 1.8 (using devenv container build/copy) and start it up:

$ docker run \
  -p 8000:8000 \
  -p 9000:9000 \
  -e OPENAI_API_KEY="$OPENAI_API_KEY" \
  -e CONTAINER_IS_RUNNING=1 \
  --rm --name temp-rag -it ghcr.io/enfold/afsoc-rag:latest "$@"
Running tasks     devenv:enterShell
Succeeded         devenv:python:uv                         234ms     
Succeeded         devenv:enterShell                        24ms      
Succeeded         rag:pythonbuild                          18503ms   
3 Succeeded                         18.74s
inside procfilescript
[nginx	] 2025/08/10 22:11:13 [warn] 325#325: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
[nginx	] nginx: [warn] could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: using the "epoll" event method
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: nginx/1.28.0
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: built by gcc 14.2.1 20250322 (GCC) 
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: OS: Linux 6.12.30
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: getrlimit(RLIMIT_NOFILE): 1024:524288
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: start worker processes
[nginx	] 2025/08/10 22:11:13 [notice] 325#325: start worker process 326
[ragserver	] WARNING:  Current configuration will not reload as not all conditions are met, please refer to documentation.
...

When I produce the same container using the same devenv setup under 1.8.1:

$ docker run \
  -p 8000:8000 \
  -p 9000:9000 \
  -e OPENAI_API_KEY="$OPENAI_API_KEY" \
  -e CONTAINER_IS_RUNNING=1 \
  --rm --name temp-rag -it ghcr.io/enfold/afsoc-rag:latest "$@"
mkdir: cannot create directory '/home': Permission denied
ln: failed to create symbolic link '/home/runner/work/afsoc-rag/afsoc-rag/.devenv/profile': No such file or directory
mkdir: cannot create directory '/run': Permission denied
ln: failed to create symbolic link '/home/runner/work/afsoc-rag/afsoc-rag/.devenv/run': No such file or directory
mkdir: cannot create directory '/home': Permission denied
Error: IoError(Custom { kind: Other, error: "Failed to initialize task cache: Database error: error returned from database: (code: 14) unable to open database file" })
mkdir: cannot create directory '/home': Permission denied
inside procfilescript
mkdir: cannot create directory '/home': Permission denied
Failed to create log directory: /run/user/1001 - mkdir /run: permission denied

To reproduce

I'm working on a repro using GH actions but it may be the case that this is any container.

Version

1.8.1

[onestardao]

Looks like a classic pre-deploy bootstrap ordering issue — container tries to run before its required init paths are available.
We’ve seen this exact failure pattern mapped as ProblemMap No.15 in our diagnostics.

If you want the full fix path (bootstrap ordering, deployment deadlock, pre-deploy collapse cases), I can share the doc. It covers why 1.8.1 triggers it while 1.8.0 didn’t, and how to make it version-safe.

[sandydoo]

Fixed in #2083?

[mcdonc]

Fixed in #2083?

Don't think so as that issue claims the bug was never relelased.

[mcdonc]

Looks like a classic pre-deploy bootstrap ordering issue — container tries to run before its required init paths are available. We’ve seen this exact failure pattern mapped as ProblemMap No.15 in our diagnostics.

If you want the full fix path (bootstrap ordering, deployment deadlock, pre-deploy collapse cases), I can share the doc. It covers why 1.8.1 triggers it while 1.8.0 didn’t, and how to make it version-safe.

I don't know what a ProblemMap is sorry, but sure, if you have any suggestions!

[sandydoo]

Fixed in #2083?

Don't think so as that issue claims the bug was never relelased.

I thought it wasn't, but it looks like it made it into 1.8.1 as the last commit.

[mcdonc]

Fixed in #2083?

Don't think so as that issue claims the bug was never relelased.

I thought it wasn't, but it looks like it made it into 1.8.1 as the last commit.

OK, cool, I'll produce some sort of multiversion test to verify

[bemyak]

I'm facing the same issue on devenv 1.8.1. I noticed that my local env vars (DEVENV_DOTFILE, DEVENV_ROOT, DEVENV_STATE, GOPATH,PATH) are leaking into the shell container. Running devenv container run shell --clean doesn't change a thing.

❯ devenv shell env | grep DEVENV_ROOT
• Using Cachix caches: devenv
✓ Building shell in 60.7ms
• Entering shell
jq: parse error: Invalid numeric literal at line 1, column 12
Running tasks     devenv:enterShell
Succeeded         devenv:enterShell                        5ms
1 Succeeded                         5.62ms
DEVENV_ROOT=/home/bemyak/path/to/my/project

❯ devenv container run shell
• Using Cachix caches: devenv
✓ Building shell container in 37.9ms

Copying container /nix/store/jbxzgbkcagk87bx0278xf3lzhh9dhz8r-image-shell.json to docker-daemon:shell:latest

Getting image source signatures
Copying blob 0dc97274a23e done   |
Copying blob 269be842d429 done   |
Copying config ad676df3df done   |
Writing manifest to image destination
• Running container shell
mkdir: cannot create directory '/home': Permission denied
ln: failed to create symbolic link '/home/bemyak/path/to/my/project/.devenv/profile': No such file or directory
ln: failed to create symbolic link '/home/bemyak/path/to/my/project/.devenv/run': No such file or directory
Error: IoError(Custom { kind: Other, error: "Failed to initialize task cache: Database error: error returned from database: (code: 14) unable to open database file" })

bash-5.3$ env | grep DEVENV_ROOT
DEVENV_ROOT=/home/bemyak/path/to/my/project

bash-5.3$ pwd
/env

So, DEVENV_ROOT is the same both inside devenv shell and container sheel, which doesn't look right.

Hope this helps!

[sandydoo]

@bemyak could you please test against main?

[bemyak]

Just tested, it seems to be fixed! Now I'm anxiously waiting for 1.8.2 😄

[sandydoo]

Fixed in 1.8.2.