ARROW-16566: [Java] Initialize JNI components on use instead of statically #13146

jackylee-ch · 2022-05-13T09:49:57Z

We are using Arrow in Spark. When speculation is turned on, it is possible for the task to be forcibly killed during the running process. It means, JniLoader.get().ensureLoaded() maybe failed with ClosedByInterruptException when the task was killed, and it should be accepted with this.
However, NativeMemoryPool use it as a static code, this leads to an init failure every time the NativeMemroyPool is trying to initialized and can never succeed. It would always fail to init NativeMemoryPool with java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool.

… failed

github-actions · 2022-05-13T09:50:20Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2022-05-13T09:52:15Z

https://issues.apache.org/jira/browse/ARROW-16566

github-actions · 2022-05-13T09:52:17Z

⚠️ Ticket has no components in JIRA, make sure you assign one.

github-actions · 2022-05-13T09:52:17Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

jackylee-ch · 2022-05-13T09:55:00Z

⚠️ Ticket has no components in JIRA, make sure you assign one.

Done

lidavidm · 2022-05-13T12:15:12Z

Could you elaborate on

When the specification is turned on

?

lidavidm · 2022-05-13T12:16:00Z

java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeMemoryPool.java

  private final long nativeInstanceId;

-  static {
+  private static void ensureLoaded() {


Is there any reason not to inline this instead?

jackylee-ch · 2022-05-13T12:31:55Z

Could you elaborate on

When the specification is turned on

?

Thanks for your attension.
When spark speculation is turned on, the spark program allows the user's logic to be executed in parelle, which means there will be two same logic running at the same time. spark will interrupt the unfinished task once one of them finished and then run other tasks.

Back to current PR, if it is killed when calling JniLoader.get.ensureLoaded in NativeMemoryPool initlization, it will cause the NativeMemoryPool to never be initialized successfully in the subsequent execution process.

pitrou · 2022-05-13T15:50:17Z

cc @lwhite1 , who has started working on various JNI aspects.

pitrou · 2022-06-07T11:47:55Z

@jackylee-ch Would you like to follow up on @lidavidm 's suggestion above?

jackylee-ch · 2022-06-07T11:57:16Z

@jackylee-ch Would you like to follow up on @lidavidm 's suggestion above?

Sure.

lidavidm

Thank you!

jackylee-ch · 2022-06-09T00:33:49Z

Thanks for your help @lidavidm @pitrou

…cally (apache#13146) We are using Arrow in Spark. When speculation is turned on, it is possible for the task to be forcibly killed during the running process. It means, `JniLoader.get().ensureLoaded()` maybe failed with `ClosedByInterruptException` when the task was killed, and it should be accepted with this. However, `NativeMemoryPool` use it as a static code, this leads to an init failure every time the NativeMemroyPool is trying to initialized and can never succeed. It would always fail to init `NativeMemoryPool` with `java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool`. Lead-authored-by: jackylee-ch <[email protected]> Co-authored-by: stczwd <[email protected]> Signed-off-by: David Li <[email protected]>

…cally (apache#13146) (#119) We are using Arrow in Spark. When speculation is turned on, it is possible for the task to be forcibly killed during the running process. It means, `JniLoader.get().ensureLoaded()` maybe failed with `ClosedByInterruptException` when the task was killed, and it should be accepted with this. However, `NativeMemoryPool` use it as a static code, this leads to an init failure every time the NativeMemroyPool is trying to initialized and can never succeed. It would always fail to init `NativeMemoryPool` with `java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool`. Lead-authored-by: jackylee-ch <[email protected]> Co-authored-by: stczwd <[email protected]> Signed-off-by: David Li <[email protected]>

…cally (apache#13146) We are using Arrow in Spark. When speculation is turned on, it is possible for the task to be forcibly killed during the running process. It means, `JniLoader.get().ensureLoaded()` maybe failed with `ClosedByInterruptException` when the task was killed, and it should be accepted with this. However, `NativeMemoryPool` use it as a static code, this leads to an init failure every time the NativeMemroyPool is trying to initialized and can never succeed. It would always fail to init `NativeMemoryPool` with `java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool`. Lead-authored-by: jackylee-ch <[email protected]> Co-authored-by: stczwd <[email protected]> Signed-off-by: David Li <[email protected]>

fix: can not init class after JniLoader.get().ensureLoaded() accepted…

4525ac5

… failed

github-actions bot added the Component: Java label May 13, 2022

jackylee-ch changed the title ~~[ARROW-16566] Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool~~ ARROW-16566: [JAVA] Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool May 13, 2022

lidavidm reviewed May 13, 2022

View reviewed changes

pitrou changed the title ~~ARROW-16566: [JAVA] Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool~~ ARROW-16566: [Java] Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool May 16, 2022

change code to inline

7e5354e

lidavidm approved these changes Jun 7, 2022

View reviewed changes

lidavidm changed the title ~~ARROW-16566: [Java] Could not initialize class org.apache.arrow.dataset.jni.NativeMemoryPool~~ ARROW-16566: [Java] Initialize JNI components on use instead of statically Jun 8, 2022

lidavidm merged commit 982960a into apache:master Jun 8, 2022

jackylee-ch deleted the fix_jni_load_failed branch June 9, 2022 00:32

lwhite1 mentioned this pull request Aug 2, 2022

Version 9.0.0 release blog post apache/arrow-site#227

Merged

ARROW-16566: [Java] Initialize JNI components on use instead of statically #13146

ARROW-16566: [Java] Initialize JNI components on use instead of statically #13146

Uh oh!

Conversation

jackylee-ch commented May 13, 2022 • edited by lidavidm Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 13, 2022

Uh oh!

github-actions bot commented May 13, 2022

Uh oh!

github-actions bot commented May 13, 2022

Uh oh!

github-actions bot commented May 13, 2022

Uh oh!

jackylee-ch commented May 13, 2022

Uh oh!

lidavidm commented May 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidavidm May 13, 2022

Choose a reason for hiding this comment

Uh oh!

jackylee-ch commented May 13, 2022

Uh oh!

pitrou commented May 13, 2022

Uh oh!

pitrou commented Jun 7, 2022

Uh oh!

jackylee-ch commented Jun 7, 2022

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

jackylee-ch commented Jun 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jackylee-ch commented May 13, 2022 •

edited by lidavidm

Loading

lidavidm commented May 13, 2022 •

edited

Loading