[ci] aarch64 python-package job failing: "cannot allocate memory in static TLS block"

[jameslamb]

Description

For the last few days, I've observed the aarch64 CI job (which we run on an x86_64 box, using QEMU for emulation), failing with errors like the following during test collection:

___________ ERROR collecting tests/python_package_test/test_basic.py ___________
ImportError while importing test module '/LightGBM/tests/python_package_test/test_basic.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/root/miniforge/envs/test-env/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/python_package_test/test_basic.py:12: in <module>
    from sklearn.datasets import dump_svmlight_file, load_svmlight_file, make_blobs
/root/miniforge/envs/test-env/lib/python3.12/site-packages/sklearn/__init__.py:97: in <module>
    from .utils._show_versions import show_versions
/root/miniforge/envs/test-env/lib/python3.12/site-packages/sklearn/utils/_show_versions.py:15: in <module>
    from ._openmp_helpers import _openmp_parallelism_enabled
E   ImportError: /root/miniforge/envs/test-env/lib/python3.12/site-packages/sklearn/utils/../../../../libgomp.so.1: cannot allocate memory in static TLS block

Reproducible example

This is happening across several different PRs, with changesets that are very unlikely to be causing this, suggesting it's some other change in the environment. For example:

• PR: [ci] [R-package] run 'R CMD check' as a foreground task #6508
• build: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16492&view=logs&j=c2f9361f-3c13-57db-3206-cee89820d5e3&t=6c24f9ed-c6ce-5b46-2a4e-317f1b2c686c

Environment info

N/A

Additional Comments

"TLS" in this error refers to "thread-local storage".

There is lots of prior discussion on similar issues:

• https://bugzilla.redhat.com/show_bug.cgi?id=1722181
• "It seems that scikit-learn has not been built correctly." Nvidia Jetson scikit-learn/scikit-learn#28362
• cannot allocate memory in static TLS block dmlc/xgboost#8488
• cannot allocate memory in static TLS block dmlc/xgboost#7110 (comment)
• ImportError: dlopen: cannot load any more object with static TLS pytorch/pytorch#2575
• aarch64: libgomp.so.1: cannot allocate memory in static TLS block opencv/opencv#14884
• Require libgomp on Linux conda-forge/scikit-learn-feedstock#220
• Start building linux-aarch64 nightlies dask-contrib/dask-sql#1144

All of those are about using libgomp on aarch64.

From https://bugzilla.redhat.com/show_bug.cgi?id=1722181:

The GNU TLS2 model which I'm afraid aarch64 uses unfortunately eats from the same TLS preallocated pool as libraries that require static TLS like libgomp, where it is performance critical to have it as static TLS.

On opencv/opencv#14884, there's some discussion about this specifically being caused by bundled libgomp in multiple Python packages, and there are suggestions that importing those libraries earlier (and therefore loading their libgomp earlier) can resolve this.

These also have some helpful information:

• https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
• https://stackoverflow.com/a/23003149/3986677
• https://stackoverflow.com/a/19468365/3986677
• https://chao-tic.github.io/blog/2018/12/25/tls

[jameslamb]

I recall now that allocation of static TLS was one of the reasons we switched x86_64 wheels to manylinux_2_28 (#5584 (comment))... there were some fixes in newer GLIBC versions related to it.

Similarly, a comment from August 2020 on https://bugzilla.redhat.com/show_bug.cgi?id=1722181 says:

the fix in glibc is allowing the installer to run correctly without the LD_PRELOAD workaround

So it's possible that we'd be less likely to see this if we moved aarch64 builds to a newer GLIBC.

They're currently using manylinux2014: https://github.com/guolinke/lightgbm-ci-docker/blob/a0df140729a4006b4ff87b880e7aa53934ecc0e7/images/manylinux2014_aarch64/Dockerfile#L1

[jameslamb]

I just tried a rebuild, and this is still happening: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16481&view=logs&j=7a417b3a-6502-5a0d-1db8-7ef6155c93de&t=380f8b13-0b2d-5f03-5de0-8353018c7351

Looking more closely at the logs, I noticed that it is happening whenever a libgomp.so is loaded... sometimes from loading scikit-learn's bundled one

E   ImportError: /root/miniforge/envs/test-env/lib/python3.12/site-packages/sklearn/utils/../../../../libgomp.so.1: cannot allocate memory in static TLS block

sometimes conda's

E   OSError: /root/miniforge/envs/test-env/bin/../lib/libgomp.so.1: cannot allocate memory in static TLS block

So I suspect part of the problem is "if you use scikit-learn, it's likely 2 libgomp.so.1s are going to be loaded"... maybe because of some change to the order of imports somewhere down the call stack. I'll keep testing changes.

[jmoralez]

Just spitballing here but since we're getting conda's libgomp, we could also try installing conda's c++ compiler in that job and that'd link against that libgomp, which is the one being loaded by numpy and scipy, and maybe that'd solve the issue. WDYT?

[jameslamb]

It looks like cuml is running into similar issues (also only in its aarch64 builds): rapidsai/cuml#5949

It has lots of the same dependencies as lightgbm (like scikit-learn, numpy, scipy, dask, and distributed from conda-forge).

---

we could also try installing conda's c++ compiler in that job and that'd link against that libgomp, which is the one being loaded by numpy and scipy

Interesting idea! Thanks for catching that conda-forge's libgomp is getting pulled into the environment... that might point to a source of the problem.

I'm nervous about using conda's compilers to build a wheel that's intended to be distributed for use outside of conda.

conda's toolchain is configured for producing conda packages, and I know it does things like rewriting paths in binaries so that conda's libraries will always be found. (not sure how much of that happens in compilers/linkers vs. in conda-build though).

BUT... that's testable, and it's worth testing. I can try that to see if it'd help.

Another thing I'm going to test... I'm going to try reducing the tests to simply import lightgbm and see if that's sufficient to trigger this problem. If it is, that'd narrow it down signficantly.

[jmoralez]

Oh sorry, I didn't realize that job was building wheels. We could also go the other way and drop conda from that job, I've used uv a lot lately and it's great.

[jameslamb]

drop conda from that job

I like that idea too! Although it could be pretty involved... I suspect conda is doing a LOT of useful environment-resolution work for us that might be difficult to replicate.

As a start... would you support commenting out this CI job for now so that we can keep making progress in the repo while we investigate this? With the understanding that this would have to be fixed and working again before we do the next release.

I'm really happy with the recent pace of non-maintainer contributions we've been getting, and I'd like to get those peoples' work merged so we build momentum with them:

• cmake: add missing lib_lightgbm_swig.so to lightgbmlib.jar on linux #6515
• Remove template parameters from copy constructor #6514
• [ci] fix some shellcheck errors in scripts #6504
• [ci] resolve shellcheck errors in .ci/check_python_dists.sh #6503
• [R-package]: add num_trees_per_iter, num_trees, and num_iter methods #6500
• [python-package] Add feature_names_in_ attribute for scikit-learn estimators (fixes #6279) #6310

@mayer79 and @nicklamiller in particular have been more patient with us than they should have to be, waiting on various CI issues 😅

[jmoralez]

would you support commenting out this CI job for now so that we can keep making progress in the repo while we investigate this?

Sure! That shouldn't block our CI since it's just a packaging problem.

[jameslamb]

put up #6517

Thankfully we now have that one macOS job still providing coverage of compiling LightGBM on aarch64.

[msarahan]

A couple of thoughts here:

1. Are you installing scikit-learn from PyPI (a wheel?) If so, is it an option to install that from conda? The conda package should not bundle libgomp, and so should not have this problem.
2. I wonder if scikit-learn has thought about name-mangling their copy of libgomp to avoid these issues. That would come at the cost of multiple copies of libgomp being loaded in one process, which may explode resource contention.

Probably no great answer right now, but all the more incentive to work on https://discuss.python.org/t/implementation-variants-rehashing-and-refocusing/54884!