[Java] Build Arrow as JPMS modules

[jduo]

Describe the enhancement requested

Allow Arrow to be referenced as JPMS modules when run in Java 9 and higher

This issue is broken down into the following sub-issues:

• [Java] Update dependencies and plugins for JPMS module support arrow#38996
• [Java] (JPMS) Add a maven plugin for compiling modules while retaining JDK8 support arrow#39134
• [Java] Build memory-core and memory-unsafe as JPMS modules arrow#38998
• [Java] Build arrow-format and arrow-vector as JPMS modules arrow#38997
• [Java] Build memory-netty as a JPMS module #150
• [Java] Build remaining modules as JPMS modules arrow#39001

The memory module related issues affect the user's command-line and may be too much of a breaking change for users.

Component(s)

Java

[FiV0]

@jduo Could you maybe explain how this new plugin of apache/arrow#39134 is supposed to be used. Since a couple of these changes have gone in, I am unable to test in Intellij. I normally went through the process described at https://arrow.apache.org/docs/developers/java/building.html#intellij, but that doesn't work anymore.

I am currently getting:

java: modules are not supported in -source 8
  (use -source 9 or higher to enable modules)

[jduo]

@jduo Could you maybe explain how this new plugin of apache/arrow#39134 is supposed to be used. Since a couple of these changes have gone in, I am unable to test in Intellij. I normally went through the process described at https://arrow.apache.org/docs/developers/java/building.html#intellij, but that doesn't work anymore.

I am currently getting:

java: modules are not supported in -source 8
  (use -source 9 or higher to enable modules)

Hi @FiV0 . I've been able to get IntelliJ working by setting it to use maven for build and test rather than using it's built-in build system.

[FiV0]

Thanks for the quick turnaround @jduo . That seems then to create a rather long feedback cycle (as it tests all of java/vector for example). Do you then create specific tests goals for every test you want to run alone or do you have some other setup which doesn't need a setup for every test?

[vibhatha]

@FiV0 what @jduo mentioned is the most convenient option. There is an alternative I use but when we use that we cannot compile the code using the IDE. By enabling Do not build before run and Do not use --module-path option in Modify Options under Run configs could also allow running test cases.

[jduo]

If I remember correctly, after you run the tests once in the IDE they'll appear in a list with results and you can run specific ones repeatedly afterwards.

[jayantsing-db]

Hi @jduo, Thanks for your work. It looks like there's still one open issue. I might be oversimplifying, but will the last open issue resolve the Arrow not compatible with JDK 16+ problem?

[jduo]

The modularization of Netty doesn't affect JDK 16. What's the particular issue you're seeing with JDK 16?

[jayantsing-db]

The modularization of Netty doesn't affect JDK 16. What's the particular issue you're seeing with JDK 16?

I came across this issue while investigating this specific comment on this PR. The comment seems to address Arrow's incompatibility with JDK 16+ (as previously mentioned in this comment). I may be missing some context, so please correct me if I’m mistaken.

[lidavidm]

Arrow is compatible with modern JDK versions. The required flags are a different problem (Arrow requires native memory and sun.misc.Unsafe is the only way to do that, e.g. see #511). We need support for FFM and also either drop support for Java <21 or find some way to achieve support for older JVMs at the same time. (#163)