[XABT] Scan for JLO needed wrappers in `LinkAssembliesNoShrink`. #9893

jpobst · 2025-03-07T18:39:02Z

Today, application builds use Cecil to scan through all Android assemblies at least twice:

Debug: _LinkAssembliesNoShrink, _GenerateJavaStubs
Release: _ILLink, _GenerateJavaStubs, _RemoveRegisterAttribute

This is a costly operation that we would like to only perform once to help speed up application builds. The long-term goal is to move the (many) steps currently performed by _GenerateJavaStubs into "linker" steps.

This PR moves the Cecil scanning required for the first step: "generating Java Callable Wrappers".

Implementation

This does not move generating JCW Java code to the linker, it only moves scanning for the JLO information needed to generate JCWs to the linker. This information is persisted as an XML file in /obj next to the processed assembly.

Example:

obj/Debug/net10.0-android/android/assets/MyApplication.dll
obj/Debug/net10.0-android/android/assets/MyApplication.jlo.xml

Doing it this way has two benefits: it helps keep the linker from getting too complex (eg: generating Java code) and it helps incremental builds. If an assembly has not changed since the last build, the JLO information in the .jlo.xml is still valid and will be reused. The assembly does not need to be re-scanned. (For example, the ~70 AndroidX assemblies that the MAUI template uses do not need to be re-scanned on every Debug build.)

The process of actually generating the JCWs is done by a new GenerateJavaCallableWrappers task in the _GenerateJavaStubs target that consumes the .jlo.xml files and outputs the .java files.

Release Builds

In an ideal Release build, we would (probably?) run this new step as part of the ILLink invocation. However there are some tricky limitations to running in the linker pipeline, chiefly being that the linker cannot load external assemblies like Java.Interop.Tool.JavaCallableWrappers.dll (and dependencies) that we need. This can be worked around by including the needed source files in the linker assembly, but this will require additional work.

Instead, for now, we will add an invocation a new _LinkAssembliesAdditionalSteps target to linked builds that runs after ILLink. This calls the existing LinkAssembliesNoShrink task to scan for JLOs but does not execute the other linker tasks that have already been run. This allows us to move forward with converting to "linker steps" for making Debug builds faster without the extra work required to move these steps into ILLink. A future enhancement can perform this work if desired. (There is also some question as to whether we actually want to create new ILLink linker steps.)

Note this temporarily leaves the $(_AndroidJLOCheckedBuild) flag that compares the generated Java files to the previous ones. This will likely be useful ensuring future moved steps are correct as well.

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

src/Xamarin.Android.Build.Tasks/Utilities/MarshalMethodsAssemblyRewriter.cs

src/Xamarin.Android.Build.Tasks/Linker/MonoDroid.Tuner/FindJavaObjectsStep.cs

jonpryor · 2025-03-11T19:30:13Z

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

@@ -1535,10 +1571,21 @@ because xbuild doesn't support framework reference assemblies.
    <_MergedManifestDocuments Condition=" '$(AndroidManifestMerger)' == 'legacy' " Include="@(ExtractedManifestDocuments)" />
  </ItemGroup>

+  <GenerateJavaCallableWrappers
+      CodeGenerationTarget="$(_AndroidJcwCodegenTarget)"
+      OutputDirectory="$(IntermediateOutputPath)android\src"


There are many concerns around incremental build performance. Three of which are:

Reducing time spent parsing the .dll files.

Reducing time spent generating the .java source

Reducing time spent compiling the .java source into .class files. (javac is slow.)

"Synchronization"

This PR addresses (1) and maybe (2) by making assembly parsing incremental: we only need to process assemblies which have changed.

Should we consider expanding upon (2), (3) and (4)?

The "Synchronization" question is rephrased as "how do we ensure that removed types are deleted"? For example, if the MyApp.dll assembly has:

class Demo : Java.Lang.Object { }

then MyApp.jlo.xml will mention Demo, and $(IntermediateOutputPath)android\src will contain crc64…/Demo.java, and $(IntermediateOutputPath)android\bin\classes will contain crc64…/Demo.class.

We then update MyApp.dll to remove the Demo type. What happens? What could or should happen?

If we don't remove src/crc64…/Demo.java and classes/crc64…/Demo.class, then we have an unsynchronized state of the world. (I think we've had previous bugs due to this; perhaps @dellis1972 remembers?)

If we delete src and classes, that means we need to regenerate .java files for everything, and recompile for everything. We maintain synchronization, but at a cost. (At least we saved on assembly parsing!)

An idea I've been kicking around (mentally and with Dean) was to have per-assembly directories. Instead of $(IntermediateOutputPath)android\src, have $(IntermediateOutputPath)android\jcw\$(AssemblyName)\src and $(IntermediateOutputPath)android\jcw\$(AssemblyName)\classes. When the assembly changes, we delete just the …\jcw\$(AssemblyName) directory, and no others. This should reduce the number of .java files we need to generate and javac invocations on incremental builds.

I don't know if this is actually viable; javac -d OUTDIR can only have one directory, so a per-assembly classes directory means we'd have N javac invocations (one per assembly) instead of 1, and then there's the question of ordering (A.dll may depend on B.dll, creating an ordering constraint on javac).

Even if per-assembly classes directories aren't viable, per-assembly src directories should be viable: delete just the per-assembly src directory and regenerate it based on the .jlo.xml file.

Which brings us to <GenerateJavaCallableWrappers/>: is there a way to make GenerateJavaCallableWrappers.OutputDirectory a per-assembly value?

Agreed that this PR only attempts to tackle (1) and that this is just a piece of the build performance puzzle. However tackling (1) is a very large effort as it also requires updating the other "_GenerateJavaStubs" pieces (marshal method rewriting, type map generation, acw map generation, android manifest merging) before we can actually get the win by removing the additional Cecil scan. As such, I think we should focus on that first.

The important note is that nothing in this PR makes those other existing performance issues worse. They remain unaffected.

Having said that and looking ahead...

Doing (2) incrementally should be ~easy, but also probably not a huge win. This step is now very cheap (<50 ms on Android template).

Doing (3) better, if possible, would be a good win. However we haven't examined it enough to know what is viable here. Even better if work here can also make D8 incremental as it is even slower than javac. (dotnet new android template FastDev build: _CompileJava: 1.83s, _CompileToDalvik: 4.07s)

(4) is just something to keep in mind for any future attempts to optimize (2) and (3). We need to ensure that incremental builds remove .java files that should no longer be generated.

I would check a dotnet new maui project to see the real cost of _CompileJava or _CompileToDalvik.

Once you bring their dependency tree of AndroidX libraries they get a lot slower.

I would check a dotnet new maui project to see the real cost of _CompileJava or _CompileToDalvik.

Yep, 16.72s and 22.94s respectively, on dotnet new maui.

@jonpryor We did have issues around deleted files not being removed when we have the enhanced fast deployment. This split the .dex files into a core runtime which was packed in the apk, and a classes.dex which was fast deployed. We had to manage classes being deleted in that scenario. All that does was stripped out though.

It should be able to calculate the list of files we expect to be generated and delete anything that is not in that list but exists in the target folder. We do something similar in Buildapk when updating an existing apk from the one aapt2 produced.

jonathanpeppers

The rest in here looks good. 👍

Is the Java.Interop bump a branch? or main?

jonathanpeppers · 2025-03-13T16:01:12Z

src/Xamarin.Android.Build.Tasks/Linker/MonoDroid.Tuner/FindJavaObjectsStep.cs

+
+		using (var sw = MemoryStreamPool.Shared.CreateStreamWriter ()) {
+			XmlExporter.Export (sw, wrappers, true);
+			Files.CopyIfStreamChanged (sw.BaseStream, destinationJLOXml);


Very minor: but we could log if it wrote the file.

jonathanpeppers · 2025-03-13T16:03:06Z

src/Xamarin.Android.Build.Tasks/Tasks/AssemblyModifierPipeline.cs

+	// Names of assemblies which don't have Mono.Android.dll references, or are framework assemblies, but which must
+	// be scanned for Java types.
+	static readonly HashSet<string> SpecialAssemblies = new HashSet<string> (StringComparer.OrdinalIgnoreCase) {
+			"Java.Interop.dll",
+			"Mono.Android.dll",
+			"Mono.Android.Runtime.dll",
+		};


Does XAJavaTypeScanner also have a list of these?

android/src/Xamarin.Android.Build.Tasks/Utilities/XAJavaTypeScanner.cs

Lines 15 to 23 in 599bd6c

class XAJavaTypeScanner

{

// Names of assemblies which don't have Mono.Android.dll references, or are framework assemblies, but which must

// be scanned for Java types.

static readonly HashSet<string> SpecialAssemblies = new HashSet<string> (StringComparer.OrdinalIgnoreCase) {

"Java.Interop.dll",

"Mono.Android.dll",

"Mono.Android.Runtime.dll",

};

Maybe they could share the list.

That part of XAJavaTypeScanner is only used for scanning the old way for the steps that haven't been migrated. It should go away once we complete the transition.

jpobst · 2025-03-13T18:27:22Z

Is the Java.Interop bump a branch? or main?

It's a branch, this PR needs dotnet/java-interop#1317 to go first.

Context: dotnet/android#9893 Create XML import/export adapters for Java Callable Wrappers (JCWs). This allows us to serialize the JCWs we find in the new `FindJavaObjectsStep` linker step (dotnet/android#9893) to disk that, and later use that serialized data in `_GenerateJavaStubs`. This file also allows us to cache the assembly scanning to disk so we don't need to rerun it on incremental builds. This format is likely a "work in progress". As we move other `Java.Lang.Object`-related scanning to linker steps we will likely find additional data they require that needs to go into this file, or that this format isn't even suitable and we use something else instead. Sample JCW XML content: <types was_scanned="true"> <type name="MainActivity" package="crc645107ba1b8b6ee4d3" application_java_class="android.app.Application" mono_runtime_initialization="mono.MonoPackageManager.LoadApplication (context);" extends_type="android.app.Activity" partial_assembly_qualified_name="tempbuild.MainActivity, tempbuild"> <constructors> <constructor name="MainActivity" method="n_.ctor:()V:" jni_signature="()V" managed_parameters="" retval="void" is_dynamically_registered="True" /> </constructors> <methods> <method name="clone" method="n_clone:()Ljava/lang/Object;:GetCloneHandler" jni_signature="()Ljava/lang/Object;" retval="java.lang.Object" /> <method name="onCreate" method="n_onCreate:(Landroid/os/Bundle;)V:GetOnCreate_Landroid_os_Bundle_Handler" jni_signature="(Landroid/os/Bundle;)V" params="android.os.Bundle p0" retval="void" super_call="p0" activate_call="p0" /> </methods> </type> </types> Most XML attributes correspond to members on `CallableWrapperType`, `CallableWrapperMethod`, etc. The XML file is a *per-assembly* data store; `//types/@was_scanned` specifies whether or not the assembly was scanned for JCWs. If the assembly did not contain JCWs (e.g. `System.Private.CoreLib.dll`), then `was_scanned` will be False.

jpobst force-pushed the dev/jpobst/generate-jcws branch 4 times, most recently from bb13f01 to d533b22 Compare March 10, 2025 08:01

jpobst marked this pull request as ready for review March 10, 2025 17:55

jpobst requested review from grendello, jonpryor, dellis1972 and jonathanpeppers as code owners March 10, 2025 17:55

jpobst mentioned this pull request Mar 10, 2025

[JavaCallableWrappers] Add XML import/export. dotnet/java-interop#1317

Merged

jonathanpeppers reviewed Mar 10, 2025

View reviewed changes

jonpryor reviewed Mar 11, 2025

View reviewed changes

jpobst force-pushed the dev/jpobst/generate-jcws branch from 804f66e to dbc94c8 Compare March 12, 2025 21:26

jpobst requested review from jonpryor and jonathanpeppers March 12, 2025 22:37

jonathanpeppers approved these changes Mar 13, 2025

View reviewed changes

jpobst added 3 commits March 17, 2025 11:51

[XABT] Scan for JLO needed wrappers in LinkAssembliesNoShrink.

22831f8

Scanned assemblies match, begin cleanup and Java output verification.

dae6453

Address review feedback.

af3be8d

jpobst force-pushed the dev/jpobst/generate-jcws branch from dbc94c8 to af3be8d Compare March 17, 2025 21:52

jpobst merged commit ea399ed into main Mar 18, 2025
58 checks passed

jpobst deleted the dev/jpobst/generate-jcws branch March 18, 2025 18:28

jpobst mentioned this pull request Mar 18, 2025

[XABT] Move scanning for ACW map JLOs to FindJavaObjectsStep. #9930

Merged

jonpryor mentioned this pull request Apr 7, 2025

.NET 10 Preview Release Notes #9571

Closed

jpobst mentioned this pull request Apr 8, 2025

[XABT] Move JLO scanning needed for typemap generation to a linker step. #10015

Merged

github-actions bot locked and limited conversation to collaborators Apr 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[XABT] Scan for JLO needed wrappers in `LinkAssembliesNoShrink`. #9893

[XABT] Scan for JLO needed wrappers in `LinkAssembliesNoShrink`. #9893

Uh oh!

jpobst commented Mar 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonpryor Mar 11, 2025

Uh oh!

jpobst Mar 11, 2025 •

edited

Loading

Uh oh!

jonathanpeppers Mar 11, 2025

Uh oh!

jpobst Mar 11, 2025

Uh oh!

dellis1972 Mar 13, 2025

Uh oh!

jonathanpeppers left a comment

Uh oh!

jonathanpeppers Mar 13, 2025

Uh oh!

jonathanpeppers Mar 13, 2025

Uh oh!

jpobst Mar 13, 2025

Uh oh!

jpobst commented Mar 13, 2025

Uh oh!

Uh oh!

Uh oh!

	class XAJavaTypeScanner
	{
	// Names of assemblies which don't have Mono.Android.dll references, or are framework assemblies, but which must
	// be scanned for Java types.
	static readonly HashSet<string> SpecialAssemblies = new HashSet<string> (StringComparer.OrdinalIgnoreCase) {
	"Java.Interop.dll",
	"Mono.Android.dll",
	"Mono.Android.Runtime.dll",
	};

﻿[XABT] Scan for JLO needed wrappers in LinkAssembliesNoShrink. #9893

﻿[XABT] Scan for JLO needed wrappers in LinkAssembliesNoShrink. #9893

Uh oh!

Conversation

jpobst commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation

Release Builds

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonpryor Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

jpobst Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

jpobst Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

dellis1972 Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers left a comment

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

jpobst Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

jpobst commented Mar 13, 2025

Uh oh!

Uh oh!

Uh oh!

[XABT] Scan for JLO needed wrappers in `LinkAssembliesNoShrink`. #9893

[XABT] Scan for JLO needed wrappers in `LinkAssembliesNoShrink`. #9893

jpobst commented Mar 7, 2025 •

edited

Loading

jpobst Mar 11, 2025 •

edited

Loading