Failing to run d8_dexbuilder with buildtools 31

[Kernald]

Description of the problem / feature request:

Using buildtools 31, Bazel can't build dex files anymore. With d8 disabled (default):

ERROR: /private/var/tmp/_bazel_kernald/dc8fa02be43b6315d36f6bf2bf056429/external/androidsdk/BUILD.bazel:13:25: Extracting interface @androidsdk//:dx_jar_import failed: missing input file 'external/androidsdk/build-tools/31.0.0/lib/dx.jar', owner: '@androidsdk//:build-tools/31.0.0/lib/dx.jar'
ERROR: /private/var/tmp/_bazel_kernald/dc8fa02be43b6315d36f6bf2bf056429/external/androidsdk/BUILD.bazel:13:25 Extracting interface @androidsdk//:dx_jar_import failed: 1 input file(s) do not exist

With d8 enabled with the following flags:

# Enable d8 merger
build --define=android_dexmerger_tool=d8_dexmerger

# Flags for the d8 dexer
build --define=android_incremental_dexing_tool=d8_dexbuilder
build --define=android_standalone_dexing_tool=d8_compat_dx
build --nouse_workers_with_dexbuilder

ERROR: /private/var/tmp/_bazel_kernald/dc8fa02be43b6315d36f6bf2bf056429/external/maven/BUILD:4748:11: Dexing external/maven/_dx/androidx_transition_transition/classes_and_libs_merged.jar_desugared.jar with applicable dexopts [] failed: (Exit 1): d8_dexbuilder failed: error executing command bazel-out/darwin-py2-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/d8_dexbuilder ... (remaining 1 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Error: Could not find or load main class com.android.tools.r8.compatdexbuilder.CompatDexBuilder
Caused by: java.lang.ClassNotFoundException: com.android.tools.r8.compatdexbuilder.CompatDexBuilder

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

In a WORKSPACE:

android_sdk_repository(
    name = "androidsdk",
    api_level = 31,
    build_tools_version = "31.0.0",
)

(I don't think the api_level actually matters here.)

Try to build anything that requires dexing.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 4.2.1

[c16a]

I'm having the same issue. Any clues what might have gone wrong? I can confirm this works with the latest hotfix of buildtools 30 as well (30.0.3)

[ahumesky]

With build tools 31, dx has been removed, and the latest version of d8 moved some classes around. I think his should fix it:
ceb498f
Can you try again with d8 and the latest bazel release? Looks like this is in 5.0 and 6.0

[alyssawilk]

Is there any advice of what to do without bumping bazel? We've just started having the same problem recently but haven't changed android_sdk_repository or our bazel version so I'm not sure why something broke?

[nkoroste]

I've managed to cherry-pick ceb498f into 4.2.2 pretty easily and it seemed to solve the problem when build_tools_version = "31.0.0", is used with d8

[alyssawilk]

Ah if anyone else ends up in our situation (where updating bazel isn't trivial) explicitly hard-coding build_tools_version to a lower version worked for us
-android_sdk_repository(name = "androidsdk", api_level = 30)
+android_sdk_repository(name = "androidsdk", api_level = 30, build_tools_version = "30.0.2")

[nkoroste]

@ahumesky @sgjesse

Good news, ceb498f fixes the issue that allows you to use build_tools_version = "31.0.0",

Bad news, the above change introduces some new issues with d8/r8 where Bazel DexMerger is falling back to Platform SDK's D8. This is causing a crash with too many methods being fit into a singular DEX.

Error: Cannot fit requested classes in a single dex file (# methods: 890463 > 65536 ; # fields: 468383 > 65536). Try supplying a main-dex list

The root cause is that D8 is being invoked with the default min-api value of 1, which forces single DEX.

It appears we should pass through —min-api to avoid this problem.

[Kernald]

@ahumesky @sgjesse

Good news, ceb498f fixes the issue that allows you to use build_tools_version = "31.0.0",

Bad news, the above change introduces some new issues with d8/r8 where Bazel DexMerger is falling back to Platform SDK's D8. This is causing a crash with too many methods being fit into a singular DEX.

Error: Cannot fit requested classes in a single dex file (# methods: 890463 > 65536 ; # fields: 468383 > 65536). Try supplying a main-dex list

The root cause is that D8 is being invoked with the default min-api value of 1, which forces single DEX.

It appears we should pass through —min-api to avoid this problem.

Another work-around is to manually passing a number of dex_shards to every android_binary. It's simple enough to compute, just a bit annoying.

[EdbertChan]

@Kernald How were you able to even get this to run?

I'm getting Error: Could not find or load main class com.android.tools.r8.compatdx.CompatDx. If you peek inside the packaged d8.jar file that is in build_tools 31.0.0, CompatDx doesn't even exist as a class. I can confirm this exists in 29.0.0 in the d8.jar. but not 31.0.0.

In fact, if you look at the R8 source, CompatDx got removed entirely. Maybe I should file this as an entirely separate issue.

[sgjesse]

CompatDx was moved from R8 to bazel com.google.devtools.build.android.r8.CompatDx. It was removed from R8 almost two years ago.

[ahumesky]

@EdbertChan what version of bazel are you running? This should be fixed with ceb498f which is in bazel 5.0.0 or newer

[EdbertChan]

@ahumesky We are running 5.1.0. We have forked it but we only have a few small diffs ontop of it mostly related to manifest merging. I am using the same configs that Kernald is using.

The error I get from --verbose_failures boils down to this line
bazel-out/k8-opt-exec-2B5CBBC6/bin/external/androidsdk/d8_compat_dx --dex '--num-threads=5' '--output=bazel-out/k8-fastbuild/bin/apps/presidio/helix/app-apk/_mobile_install/bin_debug/stub_application/classes.dex' bazel-out/k8-fastbuild/bin/apps/presidio/helix/app-apk/_mobile_install/bin_debug/stub_deploy.jar

When I look into the d8_compat_dx, I only see that a manifest file exists in it where it says it was created by "Target-Label: @androidsdk//:d8_compat_dx" and no other classes. Looking at this line in android_sdk_repository.bzl, it looks like Bazel is still trying to use the d8 packaged in build-tools in the Android SDK rather than trying to create its own.