-
Notifications
You must be signed in to change notification settings - Fork 4
Description
Description
RAPIDS 25.10 was the first release with CUDA 13 support (#208).
That support included changes to take advantage of improved fatbin compression options with nvcc 13:
- Use rapids_cuda_enable_fatbin_compression cudf#19650
- rapids-cmake Expose
rapids_cuda_enable_fatbin_compressionto have better CUDA 12/13 compression API rapids-cmake#890 - https://developer.nvidia.com/blog/whats-new-and-important-in-cuda-toolkit-13-0/#cuda_130_improves_default_compression_with_zstandard_zstd%C2%A0
As a result, for some libraries the CUDA 13 packages are now significantly smaller than the equivalent CUDA 12 libraries.
The size thresholds enforced in RAPIDS CI should be adjusted to account for this.
Benefits of this work
- ensures we're able to catch unexpected binary size increases for wheels early, when it's easier to investigate where they came from and make informed decisions about whether the size increase is acceptable
- helps keep RAPIDS wheels from growing too much, for all the reasons that's good (faster installs, lower storage footprint, reduced burden on package repositories, etc.)
Acceptance Criteria
- for all RAPIDS projects, configured size thresholds in wheel-building CI are within some small amount (25 MiB?) of the current wheel sizes
Approach
Look in the output of wheel CI for lines similar to this:
----- package inspection summary -----
file size
* compressed size: 0.5841G
* uncompressed size: 0.9001G
or pip download nightlies and check the size yourself.
For projects where the wheel sizes don't differ by CUDA major version, continue to configure the sizes in pyproject.toml
[tool.pydistcheck]
select = [
"distro-too-large-compressed",
]
# PyPI limit is 750 MiB, fail CI before we get too close to that
max_allowed_size_compressed = '675M'For others, remove that max_allowed_size_compressed setting in pyproject.toml and instead pass different values via the command line. Something like this in ci/validate_wheel.sh:
RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
if [[ "${package_dir}" == "python/libcuvs" ]]; then
if [[ "${RAPIDS_CUDA_MAJOR}" == "12" ]]; then
PYDISTCHECK_ARGS=(
--max-allowed-size-compressed '500M'
)
else
PYDISTCHECK_ARGS=(
--max-allowed-size-compressed '300M'
)
fi
else
PYDISTCHECK_ARGS=()
fi
cd "${package_dir}"
rapids-logger "validate packages with 'pydistcheck'"
pydistcheck \
--inspect \
"${PYDISTCHECK_ARGS[@]}" \
"$(echo "${wheel_dir_relative_path}"/*.whl)"(this is how these checks were configured when RAPIDS supported CUDA 11 and 12, see the diff of rapidsai/cuvs#960 for an example)
For all of them, choose a limit that is within some agreed amount (25 MiB?) of the current sizes.
Projects to check (GitHub search):
- cucim
- cudf
- cugraph
- cugraph-gnn
- cuml
- cuvs
- cuxfilter
- dask-cuda
- jupyterlab-nvdashboard
- kvikio
- nx-cugraph
- raft
- rapidsmpf
- rmm
- ucxx
Notes
Size checks were originally introduced in #110