Python command not found mooncake build script via jetson container build tensor rt llm

[zhiweit]

Search before asking

• I have searched the jetson-containers issues and found no similar feature requests.

jetson-containers Component

Packages

Bug

Ran jetson-containers build tensorrt_llm, failed at the building of mooncake docker image step. I am trying to run tensor rt llm in my jetson orin device via jetson container. I am using Jetpack 6.2.1, L4T: 36.4.4.

Error logs:

+ ldconfig
/sbin/ldconfig.real: /usr/local/cuda/targets/aarch64-linux/lib/libcusparseLt.so.0 is not a symbolic link

+ cd /opt/mooncake/
+ bash ./scripts/build_wheel.sh
+ PYTHON_VERSION=3.10
+ OUTPUT_DIR=dist
+ echo 'Building wheel for Python 3.10 with output directory dist'
Building wheel for Python 3.10 with output directory dist
Cleaning wheel-build directory
+ export LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs::/usr/local/cuda/compat:/usr/local/cuda/lib64::/usr/local/lib
+ LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs::/usr/local/cuda/compat:/usr/local/cuda/lib64::/usr/local/lib
+ echo 'Cleaning wheel-build directory'
+ rm -rf 'mooncake-wheel/mooncake_transfer_engine*'
+ rm -rf mooncake-wheel/build/
+ rm -f 'mooncake-wheel/mooncake/*.so'
Creating directory structure...
+ echo 'Creating directory structure...'
+ cp build/mooncake-integration/engine.cpython-310-aarch64-linux-gnu.so mooncake-wheel/mooncake/engine.so
+ '[' -f build/mooncake-integration/store.cpython-310-aarch64-linux-gnu.so ']'
+ echo 'Copying store.so...'
+ cp build/mooncake-integration/store.cpython-310-aarch64-linux-gnu.so mooncake-wheel/mooncake/store.so
Copying store.so...
Copying master binary...
+ echo 'Copying master binary...'
+ cp build/mooncake-store/src/mooncake_master mooncake-wheel/mooncake/
Skipping nvlink_allocator.so (not built - likely ARM64 or non-CUDA build)
Copying transfer_engine_bench...
+ '[' -f build/mooncake-transfer-engine/nvlink-allocator/nvlink_allocator.so ']'
+ echo 'Skipping nvlink_allocator.so (not built - likely ARM64 or non-CUDA build)'
+ echo 'Copying transfer_engine_bench...'
+ cp build/mooncake-transfer-engine/example/transfer_engine_bench mooncake-wheel/mooncake/
Skipping libascend_transport_mem.so (not built - Ascend disabled)
Building wheel package...
Cleaning up previous build artifacts...
+ '[' -f build/mooncake-transfer-engine/src/transport/ascend_transport/hccl_transport/ascend_transport_c/libascend_transport_mem.so ']'
+ echo 'Skipping libascend_transport_mem.so (not built - Ascend disabled)'
+ echo 'Building wheel package...'
+ cd mooncake-wheel
+ echo 'Cleaning up previous build artifacts...'
+ rm -rf dist/
+ mkdir -p dist
Installing required build packages
+ echo 'Installing required build packages'
+ pip install --upgrade pip
Using pip 25.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: https://pypi.jetson-ai-lab.io/jp6/cu126, https://pypi.jetson-ai-lab.io/jp6/cu126
Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages (25.2)
WARNING: Error parsing dependencies of devscripts: Invalid version: '2.22.1ubuntu1'
+ pip install build setuptools wheel auditwheel
Using pip 25.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: https://pypi.jetson-ai-lab.io/jp6/cu126, https://pypi.jetson-ai-lab.io/jp6/cu126
Collecting build
  Downloading https://pypi.jetson-ai-lab.io/root/pypi/%2Bf/714/5f0b5061ba90a/build-1.3.0-py3-none-any.whl (23 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (80.9.0)
Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (0.45.1)
Collecting auditwheel
  Downloading https://pypi.jetson-ai-lab.io/root/pypi/%2Bf/430/2ae79dcff242e/auditwheel-6.4.2-py3-none-any.whl (50 kB)
Requirement already satisfied: packaging>=19.1 in /usr/local/lib/python3.10/dist-packages (from build) (25.0)
Collecting pyproject_hooks (from build)
  Downloading https://pypi.jetson-ai-lab.io/root/pypi/%2Bf/9e5/c6bfa8dcc3009/pyproject_hooks-1.2.0-py3-none-any.whl (10 kB)
Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build) (2.2.1)
Collecting pyelftools>=0.24 (from auditwheel)
  Downloading https://pypi.jetson-ai-lab.io/root/pypi/%2Bf/013/df952a006db5e/pyelftools-0.32-py3-none-any.whl (188 kB)
WARNING: Error parsing dependencies of devscripts: Invalid version: '2.22.1ubuntu1'
Installing collected packages: pyelftools, pyproject_hooks, auditwheel, build
  changing mode of /usr/local/bin/auditwheel to 755
  changing mode of /usr/local/bin/pyproject-build to 755

Successfully installed auditwheel-6.4.2 build-1.3.0 pyelftools-0.32 pyproject_hooks-1.2.0
+ REPAIRED_DIR=repaired_wheels_3.10
+ mkdir -p repaired_wheels_3.10
++ uname -m
+ ARCH=aarch64
+ '[' aarch64 = aarch64 ']'
+ PLATFORM_TAG=manylinux_2_35_aarch64
+ echo 'Building for ARM64 architecture'
Building for ARM64 architecture
+ '[' 3.10 = 3.8 ']'
+ echo 'Repairing wheel with auditwheel for platform: manylinux_2_35_aarch64'
+ python -m build --wheel --outdir dist
Repairing wheel with auditwheel for platform: manylinux_2_35_aarch64
**./scripts/build_wheel.sh: line 189: python: command not found**
The command '/bin/sh -c /tmp/mooncake/install.sh || /tmp/mooncake/build.sh' returned a non-zero code: 127
[09:28:16] ===================================================================================== 
[09:28:16] ===================================================================================== 
[09:28:16] 💣 `jetson-containers build` failed after 631.3 seconds (10.5 minutes) 
[09:28:16] Error: Command 'DOCKER_BUILDKIT=0 docker build --network=host   --tag tensorrt_llm:r36.4.tegra-aarch64-cu126-22.04-mooncake   --file /home/magnus/oss-projects/jetson-containers/packages/llm/dynamo/mooncake/Dockerfile   --build-arg BASE_IMAGE=tensorrt_llm:r36.4.tegra-aarch64-cu126-22.04-torchao   --build-arg MOONCAKE_VERSION="0.3.7"   --build-arg MOONCAKE_VERSION_SPEC="0.3.7"   --build-arg IS_SBSA="False"    /home/magnus/oss-projects/jetson-containers/packages/llm/dynamo/mooncake 2>&1 | tee /home/magnus/oss-projects/jetson-containers/logs/20250922_091745/build/32o34_tensorrt_llm_r36.4.tegra-aarch64-cu126-22.04-mooncake.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 127. 
[09:28:16] ===================================================================================== 
[09:28:16] ===================================================================================== 
[09:28:16] Failed building:  tensorrt_llm

Environment

Jetpack 6.2.1

• Model: NVIDIA Jetson AGX Orin Developer Kit
• L4T: 36.4.4
  NV Power Mode[0]: MAXN
  Hardware:
• P-Number: p3701-0005
• Module: NVIDIA Jetson AGX Orin (64GB ram)
  Platform:
• Distribution: Ubuntu 22.04 Jammy Jellyfish
• Release: 5.15.148-tegra
  jtop:
• Version: 4.3.2
• Service: Active
  Libraries:
• CUDA: 12.6.68
• cuDNN: 9.3.0.75
• TensorRT: 10.3.0.30
• VPI: 3.2.4
• Vulkan: 1.3.204
• OpenCV: 4.8.0 - with CUDA: NO

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!