GH-35245: [Java][Dataset][Linux] Enable GCS #35246

henrymai · 2023-04-20T07:39:08Z

Rationale for this change

Enables GCS when building the Arrow Dataset for Java and also fixes various java build failures.

Currently we are using our own custom Arrow Dataset build with GCS turned on, but we would rather this be enabled in the official releases from Arrow.

GCS support is already enabled for cpp, python, ruby, python, and r already, so there should be no reason not to enable this on java as well.

What changes are included in this PR?

Changes to enable GCS for Java Arrow Dataset on just Linux for now.
Fixes to flight-sql-jdbc-driver/pom.xml. Without these fixes the flight-sql-jdbc-driver build will fail with the following errors:

    [WARNING] Used undeclared dependencies found:
    [WARNING]    org.bouncycastle:bcpkix-jdk15on:jar:1.61:runtime
    [WARNING]    org.apache.arrow:arrow-memory-core:jar:12.0.0-SNAPSHOT:runtime
    [WARNING]    org.hamcrest:hamcrest:jar:2.2:runtime
    [WARNING]    org.apache.arrow:flight-sql:jar:12.0.0-SNAPSHOT:runtime
    [WARNING]    org.mockito:mockito-core:jar:2.25.1:test
    [WARNING]    org.apache.arrow:flight-core:jar:12.0.0-SNAPSHOT:runtime
    [WARNING]    org.slf4j:slf4j-api:jar:1.7.25:runtime
    [WARNING]    io.netty:netty-common:jar:4.1.82.Final:runtime
    [WARNING]    joda-time:joda-time:jar:2.10.14:runtime
    [WARNING]    org.apache.calcite.avatica:avatica:jar:1.18.0:runtime
    [WARNING]    com.google.protobuf:protobuf-java:jar:3.21.6:runtime
    [WARNING]    org.apache.arrow:arrow-vector:jar:12.0.0-SNAPSHOT:runtime
    [WARNING]    com.google.guava:guava:jar:31.1-jre:runtime

    [...]

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.0.1:analyze-only (analyze) on project flight-sql-jdbc-driver: Dependency problems found -> [Help 1]

    Caused by: java.lang.NullPointerException: Could not find test data path. Set the environment variable ARROW_TEST_DATA or the JVM property arrow.test.dataRoot.
      at java.util.Objects.requireNonNull(Objects.java:228)
      at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.getTestDataRoot(FlightSqlTestCertificates.java:40)
      at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.getFlightTestDataRoot(FlightSqlTestCertificates.java:51)
      at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.exampleTlsCerts(FlightSqlTestCertificates.java:60)
      at org.apache.arrow.driver.jdbc.ConnectionTlsTest.<clinit>(ConnectionTlsTest.java:59)

Are these changes tested?

I've tested the build by running:

$HOME/.local/bin/archery docker run java-jni-manylinux-2014

I've also tested the resulting ./java/dataset/target/arrow-dataset-12.0.0-SNAPSHOT.jar from running the command and have verified that GCS support is enabled.

Are there any user-facing changes?

Yes, Java Arrow Dataset will now work with GCS.

Closes: [Java][Linux] Enable GCS for Arrow Dataset #35245

github-actions · 2023-04-20T07:39:33Z

Closes: [Java][Linux] Enable GCS for Arrow Dataset #35245

github-actions · 2023-04-20T07:39:35Z

⚠️ GitHub issue #35245 has been automatically assigned in GitHub to PR creator.

kou · 2023-04-20T07:58:54Z

ci/scripts/java_jni_macos_build.sh

Could you keep this list in alphabetical order?

Also I just noticed that this is out of order (before my change):

: ${ARROW_RPATH_ORIGIN:=ON} : ${ARROW_ORC:=ON}

ci/scripts/java_jni_macos_build.sh

ci/scripts/java_jni_manylinux_build.sh

kou · 2023-04-20T08:01:37Z

@github-actions crossbow submit java-jars

github-actions · 2023-04-20T08:04:11Z

Revision: 9f4d80c8b0239aeee6a85234832b0b2d29a5dc46

Submitted crossbow builds: ursacomputing/crossbow @ actions-f4323dd042

Task	Status
java-jars

henrymai · 2023-04-20T09:01:15Z

Looks like the macOS builds are failing (which I kind of expected).
I'll remove the macOS changes from my patch and let someone else that has a Mac take that on in a follow on patch.

Enables GCS when building the Arrow Dataset for Java and also fixes various java build failures. Without the changes to flight-sql-jdbc-driver/pom.xml the flight-sql-jdbc-driver build will fail with the following errors: [WARNING] Used undeclared dependencies found: [WARNING] org.bouncycastle:bcpkix-jdk15on:jar:1.61:runtime [WARNING] org.apache.arrow:arrow-memory-core:jar:12.0.0-SNAPSHOT:runtime [WARNING] org.hamcrest:hamcrest:jar:2.2:runtime [WARNING] org.apache.arrow:flight-sql:jar:12.0.0-SNAPSHOT:runtime [WARNING] org.mockito:mockito-core:jar:2.25.1:test [WARNING] org.apache.arrow:flight-core:jar:12.0.0-SNAPSHOT:runtime [WARNING] org.slf4j:slf4j-api:jar:1.7.25:runtime [WARNING] io.netty:netty-common:jar:4.1.82.Final:runtime [WARNING] joda-time:joda-time:jar:2.10.14:runtime [WARNING] org.apache.calcite.avatica:avatica:jar:1.18.0:runtime [WARNING] com.google.protobuf:protobuf-java:jar:3.21.6:runtime [WARNING] org.apache.arrow:arrow-vector:jar:12.0.0-SNAPSHOT:runtime [WARNING] com.google.guava:guava:jar:31.1-jre:runtime [...] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.0.1:analyze-only (analyze) on project flight-sql-jdbc-driver: Dependency problems found -> [Help 1] And also fail with: Caused by: java.lang.NullPointerException: Could not find test data path. Set the environment variable ARROW_TEST_DATA or the JVM property arrow.test.dataRoot. at java.util.Objects.requireNonNull(Objects.java:228) at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.getTestDataRoot(FlightSqlTestCertificates.java:40) at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.getFlightTestDataRoot(FlightSqlTestCertificates.java:51) at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.exampleTlsCerts(FlightSqlTestCertificates.java:60) at org.apache.arrow.driver.jdbc.ConnectionTlsTest.<clinit>(ConnectionTlsTest.java:59)

lidavidm

Thanks. The Java changes look fine. We can punt on macOS for now. @davisusanibar might you be able to follow up there?

kou · 2023-04-20T11:38:31Z

@github-actions crossbow submit java-jars

github-actions · 2023-04-20T11:41:14Z

Revision: 98d47c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-b74b79c10c

Task	Status
java-jars

henrymai · 2023-04-20T13:47:45Z

It looks like there are two unrelated test failures:

kou · 2023-04-20T13:55:11Z

Yes. They are unrelated.

Could you check the built artifacts at https://github.com/ursacomputing/crossbow/releases/tag/actions-b74b79c10c-github-java-jars ?

davisusanibar · 2023-04-20T14:29:20Z

Thanks. The Java changes look fine. We can punt on macOS for now. @davisusanibar might you be able to follow up there?

Sure, let me also consider Windows changes needed.

henrymai · 2023-04-20T14:31:43Z

Yes. They are unrelated.

Could you check the built artifacts at https://github.com/ursacomputing/crossbow/releases/tag/actions-b74b79c10c-github-java-jars ?

Works fine. See this screenshot for proof:

For context, I just replaced our existing custom 10.0.1 build with the 12.0.1-SNAPSHOT from the built artifacts link that you sent and verified it with a simple test to a public gcs repo.

kou

+1

Thanks.

henrymai · 2023-04-20T14:38:30Z

Thanks @kou @lidavidm and @davisusanibar for the very fast turn around.

davisusanibar · 2023-04-21T20:31:45Z

Enables GCS when building the Arrow Dataset for Java and also fixes various java build failures.

MacOSX changes needed: GH-35274: [Java][CI] Enable GCS on MacOS #35277
Windows OSX: The changes in the Windows OS are taking me a long time to test, I will handle it in another PR

### Rationale for this change Enables GCS when building the Arrow Dataset for Java and also fixes various java build failures. Currently we are using our own custom Arrow Dataset build with GCS turned on, but we would rather this be enabled in the official releases from Arrow. GCS support is already enabled for cpp, python, ruby, python, and r already, so there should be no reason not to enable this on java as well. ### What changes are included in this PR? - Changes to enable GCS for Java Arrow Dataset on just Linux for now. - Fixes to flight-sql-jdbc-driver/pom.xml. Without these fixes the flight-sql-jdbc-driver build will fail with the following errors: ``` [WARNING] Used undeclared dependencies found: [WARNING] org.bouncycastle:bcpkix-jdk15on:jar:1.61:runtime [WARNING] org.apache.arrow:arrow-memory-core:jar:12.0.0-SNAPSHOT:runtime [WARNING] org.hamcrest:hamcrest:jar:2.2:runtime [WARNING] org.apache.arrow:flight-sql:jar:12.0.0-SNAPSHOT:runtime [WARNING] org.mockito:mockito-core:jar:2.25.1:test [WARNING] org.apache.arrow:flight-core:jar:12.0.0-SNAPSHOT:runtime [WARNING] org.slf4j:slf4j-api:jar:1.7.25:runtime [WARNING] io.netty:netty-common:jar:4.1.82.Final:runtime [WARNING] joda-time:joda-time:jar:2.10.14:runtime [WARNING] org.apache.calcite.avatica:avatica:jar:1.18.0:runtime [WARNING] com.google.protobuf:protobuf-java:jar:3.21.6:runtime [WARNING] org.apache.arrow:arrow-vector:jar:12.0.0-SNAPSHOT:runtime [WARNING] com.google.guava:guava:jar:31.1-jre:runtime [...] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.0.1:analyze-only (analyze) on project flight-sql-jdbc-driver: Dependency problems found -> [Help 1] ``` ``` Caused by: java.lang.NullPointerException: Could not find test data path. Set the environment variable ARROW_TEST_DATA or the JVM property arrow.test.dataRoot. at java.util.Objects.requireNonNull(Objects.java:228) at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.getTestDataRoot(FlightSqlTestCertificates.java:40) at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.getFlightTestDataRoot(FlightSqlTestCertificates.java:51) at org.apache.arrow.driver.jdbc.utils.FlightSqlTestCertificates.exampleTlsCerts(FlightSqlTestCertificates.java:60) at org.apache.arrow.driver.jdbc.ConnectionTlsTest.<clinit>(ConnectionTlsTest.java:59) ``` ### Are these changes tested? I've tested the build by running: ``` $HOME/.local/bin/archery docker run java-jni-manylinux-2014 ``` I've also tested the resulting `./java/dataset/target/arrow-dataset-12.0.0-SNAPSHOT.jar` from running the command and have verified that GCS support is enabled. ### Are there any user-facing changes? Yes, Java Arrow Dataset will now work with GCS. * Closes: apache#35245 Authored-by: Henry Mai <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

henrymai requested review from assignUser, kou, lidavidm and raulcd as code owners April 20, 2023 07:39

github-actions bot added Component: Java awaiting review Awaiting review labels Apr 20, 2023

kou reviewed Apr 20, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Apr 20, 2023

henrymai force-pushed the enable_gcs branch from 9f4d80c to d02fdbd Compare April 20, 2023 08:33

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 20, 2023

henrymai force-pushed the enable_gcs branch from d02fdbd to d9b589a Compare April 20, 2023 09:04

henrymai changed the title ~~GH-35245: [Java][Arrow Dataset] Enable GCS~~ GH-35245: [Java][Arrow Dataset][Linux] Enable GCS Apr 20, 2023

henrymai force-pushed the enable_gcs branch from d9b589a to 98d47c5 Compare April 20, 2023 09:04

lidavidm reviewed Apr 20, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Apr 20, 2023

kou changed the title ~~GH-35245: [Java][Arrow Dataset][Linux] Enable GCS~~ GH-35245: [Java][Dataset][Linux] Enable GCS Apr 20, 2023

kou approved these changes Apr 20, 2023

View reviewed changes

kou merged commit 7ea2d98 into apache:main Apr 20, 2023

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Apr 20, 2023

henrymai deleted the enable_gcs branch April 20, 2023 14:38

GH-35245: [Java][Dataset][Linux] Enable GCS #35246

GH-35245: [Java][Dataset][Linux] Enable GCS #35246

Uh oh!

Conversation

henrymai commented Apr 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Apr 20, 2023

Uh oh!

github-actions bot commented Apr 20, 2023

Uh oh!

kou Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

henrymai Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

henrymai Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou commented Apr 20, 2023

Uh oh!

github-actions bot commented Apr 20, 2023

Uh oh!

henrymai commented Apr 20, 2023

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

kou commented Apr 20, 2023

Uh oh!

github-actions bot commented Apr 20, 2023

Uh oh!

henrymai commented Apr 20, 2023

Uh oh!

kou commented Apr 20, 2023

Uh oh!

davisusanibar commented Apr 20, 2023

Uh oh!

henrymai commented Apr 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

henrymai commented Apr 20, 2023

Uh oh!

davisusanibar commented Apr 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

henrymai commented Apr 20, 2023 •

edited

Loading

henrymai commented Apr 20, 2023 •

edited

Loading