Warning - All configs were filtered out because none of them sufficiently match the hints.

[adam-hartshorne]

Description

When running any jax code with v0.6.1+, that ran fine up to and including v0.6.0, I now receive the following warming message many times at start-up,

W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.

This is on a linx box running Ubuntu 24.04, Nvidia Driver 570.152.03 and Cuda 12.8

System info (python version, jaxlib version, accelerator, etc.)

Jax / Jaxlib - 0.6.1+
Ubuntu - 24.04
Nvidia Driver - 570.152.03
Cuda - 12.8

[ben0i0d]

The same issue was also found on 7900XTX（ROCM 6.4.1）, Debian 12, and JAX 0.6.3.dev.

[ben0i0d]

For AMD GPUs, I resolved this issue by using the latest 0.6.0 release from https://github.com/ROCm/jax/releases.

The build command I used was:

python ./build/build.py build --wheels=jax-rocm-plugin,jax-rocm-pjrt --rocm_version=60 --rocm_path=/opt/rocm-6.4.1

Then, simply install JAX and jaxlib version 0.6.0.
Note: I installed JAX using the source code tarball provided in the release.

I have no knowledge as to why it is unavailable in the higher versions.

[danielpmorton]

Also noticing this on the following machine:
Ubuntu 24.04.2
Nvidia Driver Version: 570.133.07
CUDA Version: 12.8

[carlosgmartin]

Same issue:

$ py -c "import jax; jax.print_environment_info()"
jax:    0.6.2
jaxlib: 0.6.2
numpy:  1.26.4
python: 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
device info: NVIDIA A100-SXM4-40GB-1, 1 local devices"
process_count: 1
platform: uname_result(system='Linux', node='marvel-1-13', release='4.18.0-372.32.1.el8_6.x86_64', version='#1 SMP Fri Oct 28 15:56:52 EDT 2022', machine='x86_64')

[Yongda1]

The same issue:
jax: 0.6.3

[ben0i0d]

Here's one strange thing to add:
when I was using version 0.6.3 and the model was using bfloat16, I didn't encounter this issue.

Below are my settings:（7900xtx）

jax:    0.6.3.dev20250708
jaxlib: 0.6.3.dev20250708
numpy:  2.3.1
python: 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0]
device info: AMD Radeon Graphics-1, 1 local devices"
process_count: 1
platform: uname_result(system='Linux', node='CPI-LW', release='6.1.0-37-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.1.140-1 (2025-05-22)', machine='x86_64')

[skylooop]

Same issue. Newest jax, ubuntu 22.04, cuda toolkit 12.8, nvidia-driver 570

[mohammedbelgoumri]

Same problem

jax: 0.6.2
jaxlib: 0.6.2
python: 3.12.11
Cuda: 12.9
Nvidia Driver: 575.64.05
OS: NixOS 25.11

[dmacd]

Just upgraded from jax-0.5, same problem:

jax-0.6.2 
jax-cuda12-pjrt-0.6.2 
jax-cuda12-plugin-0.6.2 
jaxlib-0.6.2 nvidia-cudnn-cu12-9.13.1.26 
nvidia-nvshmem-cu12-3.4.5
Driver Version: 535.129.03   CUDA Version: 12.2