ci: rework tests to break out ios/android tests

[henryiii]

Fail fast disabled, and trying to split out the android/ios runs.

The total time shouldn't be that much more than before, need to verify (a minute or two on sample builds is fine).

[joerick]

What do you think about also introducing flaky on the ios tests?

[henryiii]

We could, or there's an action that reruns on failure; actually I see at least two:

• https://github.com/marketplace/actions/retry-step
• https://github.com/marketplace/actions/retry-action

I've used one of those before, not sure which one.

[Copilot]

Pull Request Overview

This PR restructures the CI test workflow to separate iOS and Android tests into dedicated test runs, while maintaining existing coverage for other platforms. The changes enable platform-specific testing by introducing a platform matrix parameter and updating test configuration to handle different build frontends.

• Split test matrix to explicitly separate iOS/Android from native platform tests
• Updated test fixtures to intercept build calls for Android and iOS platforms
• Modified CI workflow to conditionally run certain steps based on platform type

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
unit_test/main_tests/conftest.py	Added Android and iOS platform imports and build interception for testing
.github/workflows/test.yml	Restructured test matrix to separate platform-specific runs and added conditional logic for different platforms

[henryiii]

Weird, android is getting stuck in the unit_tests and hanging, while that should be identical to native. And macOS android is failing to download. Not sure what is causing this yet, though.

[joerick]

The android unit test is probably failing due to something related to Docker - the docker tests are selected on that runner but binfmt isn't installed. I think my suggestion would be to not run the docker unit tests when testing android.

I haven't been able to figure out why the macOS android build is failing on Azure though.

[joerick]

@mhsmith there's a test failure here that you might be interested in? On Azure pipelines, macOS x86_64, the android runner seems to fail while installing the android ndk. The error looks like this:

2025-08-11T09:49:21.8926460Z > assert "Android requires the build frontend to be 'build'" in capfd.readouterr().err 2025-08-11T09:49:21.8942480Z E assert "Android requires the build frontend to be 'build'" in 'Installing NDK - this may take several minutes\nWarning: An error occurred while preparing SDK package NDK (Side by side) 27.2.12479018: /Users/runner/Library/Android/sdk/.temp/PackageOperation01/unzip/android-ndk-r27c/sources/third_party/shaderc/third_party/spirv-tools/source/fuzz/fuzzer_pass_push_ids_through_variables.cpp.\ncibuildwheel: \x1b[91merror\x1b[0m: Command [\'/private/var/folders/hg/d8rl25qs4nl00s45_rtwkjmr0000gn/T/cibw-run-ho0e8esi/cp313-android_x86_64/python/android.py\', \'env\'] failed with code 1. Loading package information... \nLoading local repository... \n[ ] 3% Loading local repository... \n[ ] 3% Fetch remote repository... \n[= ] 3% Fetch remote repository... \n[= ] 4% Fetch remote repository... \n[= ] 5% Fetch remote repository... \n[== ] 5% Fetch remote repository... \n[== ] 6% Fetch remote repository... \n[== ] 7% Fetch remote repository... \n[== ] 7% Computing updates... \n[=== ] 8% Computing updates... \n[=== ] 10% Computing updates... \n[=== ] 10% Installing NDK (Side by side) 27.2\n[=== ] 10% Verifying previous download... \n[=== ] 10% Checking existing file... \n[=== ] 10% Downloading android-ndk-r27c-darwi\n[==== ] 10% Downloading android-ndk-r27c-darwi\n[==== ] 11% Downloading android-ndk-r27c-darwi\n[==== ] 12% Downloading android-ndk-r27c-darwi\n[===== ] 13% Downloading android-ndk-r27c-darwi\n[===== ] 14% Downloading android-ndk-r27c-darwi\n[===== ] 15% Downloading android-ndk-r27c-darwi\n[====== ] 15% Downloading android-ndk-r27c-darwi\n[====== ] 16% Downloading android-ndk-r27c-darwi\n[====== ] 17% Downloading android-ndk-r27c-darwi\n[======= ] 18% Downloading android-ndk-r27c-darwi\n[======= ] 19% Downloading android-ndk-r27c-darwi\n[======= ] 20% Downloading android-ndk-r27c-darwi\n[======== ] 20% Downloading android-ndk-r27c-darwi\n[======== ] 21% Downloading android-ndk-r27c-darwi\n[======== ] 22% Downloading android-ndk-r27c-darwi\n[========= ] 23% Downloading android-ndk-r27c-darwi\n[========= ] 24% Downloading android-ndk-r27c-darwi\n[========= ] 25% Downloading android-ndk-r27c-darwi\n[========== ] 25% Downloading android-ndk-r27c-darwi\n[========== ] 26% Downloading android-ndk-r27c-darwi\n[========== ] 27% Downloading android-ndk-r27c-darwi\n[=========== ] 28% Downloading android-ndk-r27c-darwi\n[=========== ] 29% Downloading android-ndk-r27c-darwi\n[=========== ] 30% Downloading android-ndk-r27c-darwi\n[============ ] 30% Downloading android-ndk-r27c-darwi\n[============ ] 31% Downloading android-ndk-r27c-darwi\n[============ ] 32% Downloading android-ndk-r27c-darwi\n[============ ] 33% Downloading android-ndk-r27c-darwi\n[============ ] 33% Unzipping... \n[============ ] 33% Unzipping... android-ndk-r27c/ \n[============ ] 33% Unzipping... android-ndk-r27c/simp\n[============= ] 33% Unzipping... android-ndk-r27c/simp\n[============= ] 33% Unzipping... android-ndk-r27c/ndk-\n[============= ] 33% Unzipping... android-ndk-r27c/wrap\n[============= ] 33% Unzipping... android-ndk-r27c/meta\n[============= ] 33% Unzipping... android-ndk-r27c/pyth\n[============= ] 33% Unzipping... android-ndk-r27c/sour\n[============= ] 33% Unzipping... android-ndk-r27c/preb\n[============= ] 33% Unzipping... android-ndk-r27c/READ\n[============= ] 33% Unzipping... android-ndk-r27c/sour\n[===================== ] 55% Unzipping... android-ndk-r27c/sour\n \n[===================== ] 55% Unzipping... android-ndk-r27c/sour\nCommand "set -eu; HOST=x86_64-linux-android; PREFIX=/private/var/folders/hg/d8rl25qs4nl00s45_rtwkjmr0000gn/T/cibw-run-ho0e8esi/cp313-android_x86_64/python/prefix; . /private/var/folders/hg/d8rl25qs4nl00s45_rtwkjmr0000gn/T/cibw-run-ho0e8esi/cp313-android_x86_64/python/android-env.sh; export" returned exit status 1\n\n\n'

It seems to just stop unzipping the archive and error out. The output's a bit cluttered because pytest is capturing it, I'll try to make it easier to read...

[mhsmith]

Could the runner be running out of disk space while unzipping? I hit that problem on GHA, though I think the symptoms were different. I worked around it here:

cibuildwheel/.github/workflows/test.yml

Lines 74 to 75 in 3ebb6bd

	- name: Free up disk space
	if: runner.os == 'Linux'

[joerick]

Ah, it looks like I might have accidentally worked around it! I moved the first android test into the serial phase of the tests and it disappeared. So I'm thinking that maybe the issue was that the sdkmanager call was being performed concurrently, and that was failing in a weird way. Maybe the android-env.sh script could lock around this, as it is system global state. I'm actually a bit surprised that sdkmanager doesn't do that itself.

[mhsmith]

Ah, this might be because the NDK version specified in the android-env file is no longer pre-installed on the runners, so the android-env file now needs to install it. The NDK version will be updated in the next release of Python 3.13, but the versions on the runners change frequently, so the tests should be robust enough to cope with it not being present.

It should be a sufficient workaround to run at least one Android test in the serial stage that gets as far as checking the NDK installation. This wasn't previously happening on runners that don't support the emulator, because the serial tests all require the emulator and were therefore skipped.

Now that you've moved another test into the serial phase, it should be possible to remove the flaky decorator, unless it was there for some other reason.

[Copilot]

Pull Request Overview

This PR restructures the CI pipeline to separate Android and iOS tests from native platform tests, enabling parallel execution while maintaining fail-fast disabled behavior. The changes split the test matrix to run Android/iOS tests on specific platforms while keeping native tests running on all supported platforms.

Key changes:

• Test matrix separation for Android/iOS vs native platform tests
• Modified test runner to accept --test-select parameter for targeted test execution
• Reorganized Android test ordering to ensure flaky initial setup test runs first

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

File	Description
azure-pipelines.yml	Restructured job matrix to separate Android/iOS tests from native tests with conditional configurations
.github/workflows/test.yml	Updated test matrix to include separate Android/iOS test runs with modified build frontend and platform settings
test/test_android.py	Moved and marked the expected wheels test as serial and flaky to ensure proper test ordering

[joerick]

Nice, I think this is a great improvement to our CI setup.