Installation of Exo (clustering software) fails inside Jetson Docker container

[satoutahhaithem]

Search before asking

• I have searched the jetson-containers issues and found no similar feature requests.

jetson-containers Component

Docker

Bug

Problem
I first tried installing Exo on my native Jetson AGX Xavier (Ubuntu 22.04, aarch64) using my Jetson-aware fork (https://github.com/satoutahhaithem/exo_for_jetson_with_torch). After patching the source to detect the Jetson platform, pip install -e . still fails due to PyTorch not offering a compatible ARM64 wheel.

To work around this, I then launched the official dusty-nv/jetson-containers Docker image (Ubuntu 22.04, aarch64). Inside the container, installation now stalls on a strict OpenCV version pin:

Actual behavior

Native: PyTorch fails to install on ARM64.

Container: OpenCV wheel 4.10.0.84 is unavailable for aarch64, causing pip to error.

Expected behavior
Exo should install seamlessly on Jetson (native or Docker) with ARM64-compatible PyTorch and OpenCV.

Steps to reproduce
git clone https://github.com/satoutahhaithem/exo_for_jetson_with_torch
cd exo_for_jetson_with_torch/Exo-pytorch/exo-pt
python3 -m venv v && source v/bin/activate
pip install -e .

Jetson Docker

docker run --rm -it nvcr.io/nvidia/l4t-base:r35.2.1
apt-get update && apt-get install -y python3.10-venv
python3 -m venv v && source v/bin/activate
git clone https://github.com/satoutahhaithem/exo_for_jetson_with_torch
cd exo_for_jetson_with_torch/Exo-pytorch/exo-pt
pip install -e .

Environment

ERROR: Cannot install exo==0.0.1 because these package versions have conflicting dependencies.
The conflict is caused by:
exo 0.0.1 depends on opencv-python==4.10.0.84
The user requested (constraint) opencv-python==4.10.0
...
ERROR: ResolutionImpossible
Has anyone successfully installed Exo in this container? Any suggestions—such as modifying version pins or source tweaks—would be greatly appreciated.

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[xxnuo]

A fork to support jetson, using it in exo way
https://github.com/xxnuo/xotorch

But I ran the large model qwen2.5 0.5b qwen3 32b qwen3 30b a3b on two Jetson AGX Orins and found that it can communicate normally but there is no output. If you have any progress, please comment here, thank you.

[satoutahhaithem]

Yep, it also works on my Jetson, but the problem is that it doesn’t give the expected performance. For me, it only reaches about 5 tokens/s. We tested it with Xotorch, like you, together with the founder of this repository, but the results were still not good.

This is related to the issue mentioned in exo-explore/exo#834. As for the current situation, on Linux there are still no good results (whether using Tinygrad or Torch); it only works well with MLX. There might be a solution using MLX because some people are now developing mlx-cuda for Linux in the mlx repository( we can use it as an inference engine in exo), which would allow us to run MLX on Linux. However, there are still some problems since this is still under development — for example, new commits were made just yesterday.

For this reason, I also opened an issue in the Exo repository to request mlx-cuda support (exo-explore/exo#861). But another problem is that the Exo team has stopped actively developing this repository.

[xxnuo]

Yes, the exo team has terminated the development of exo. Other frameworks that support distributed infer seem to be vllm and deepspeed.

But they seem to be basically unusable on jetson? If you know anything that can make jetson infer on multiple boards, please let me know, thanks, I want to give it a try.

[satoutahhaithem]

Yes, actually, the main problem occurs on all Linux machines: we don’t get good performance. This means the main issue is with the inference engine, because Exo uses Tinygrad by default, which is not very efficient. We also tested PyTorch, but the results were still not good. However, on Apple machines, it works well.

To solve this problem, we need to use MLX on Linux machines(works only in apple machine before). Previously, this was impossible, but in recent weeks MLX has developed support for Linux machines using CUDA as a backend. Here is the pull request: ml-explore/mlx#1983.

I personally tested MLX on a machine with CUDA 12.8 (https://github.com/satoutahhaithem/mlx_cuda_backend_tests). MLX works, though it is still a little unstable, but it’s a good starting point.

Now, we need to adapt it to the Exo code to support sharding and integrate MLX-CUDA. This way, we will be able to use MLX inside Exo. I’m working on it, and if there are any updates, I will let you know.