Skip to content

Conversation

@kou
Copy link
Member

@kou kou commented Oct 8, 2025

Rationale for this change

There are link errors with build options for JNI on macOS.

What changes are included in this PR?

ARROW_BUNDLED_STATIC_LIBS has CMake target names defined in Apache Arrow not find_package()-ed target names. So we should use aws-c-common not AWS::aws-c-common.

Recent aws-c-common or something use the Network framework. So add Network to Arrow::arrow_bundled_dependencies dependencies.

Don't use compute/kernels/temporal_internal.cc in libarrow.dylib and libarrow_compute.dylib to avoid duplicated symbols error.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

@github-actions
Copy link

github-actions bot commented Oct 8, 2025

⚠️ GitHub issue #47748 has been automatically assigned in GitHub to PR creator.

Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is beyond my comprehension, but is this because of a change we have introduced recently?
I am happy for this to be merged, just trying to understand :)

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 8, 2025
@kou
Copy link
Member Author

kou commented Oct 9, 2025

No... It was introduced in 2022...: #13911

BTW, this was not enough:

https://github.com/kou/arrow-java/actions/runs/18334827827/job/52217628114#step:12:2474

  : && /usr/bin/c++ -O3 -DNDEBUG -arch arm64 -mmacosx-version-min=14.0 -dynamiclib -Wl,-headerpad_max_install_names -o dataset/libarrow_dataset_jni.dylib -install_name @rpath/libarrow_dataset_jni.dylib dataset/CMakeFiles/arrow_java_jni_dataset.dir/src/main/cpp/jni_wrapper.cc.o dataset/CMakeFiles/arrow_java_jni_dataset.dir/src/main/cpp/jni_util.cc.o  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_dataset.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_substrait.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_dataset.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libparquet.a  /opt/homebrew/Cellar/thrift/0.22.0/lib/libthrift.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_acero.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_compute.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_bundled_dependencies.a  -framework CoreFoundation  -framework Security  /opt/homebrew/opt/openssl@3/lib/libssl.a  /opt/homebrew/opt/openssl@3/lib/libcrypto.a  /opt/homebrew/opt/lz4/lib/liblz4.a  /opt/homebrew/lib/libsnappy.a  /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libz.tbd  /opt/homebrew/lib/libzstd.a  /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libcurl.tbd  /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libcurl.tbd  /opt/homebrew/lib/libutf8proc.a && :
  ld: warning: ignoring duplicate libraries: '/Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_dataset.a'
  Undefined symbols for architecture arm64:
    "__nw_content_context_default_message", referenced from:
        _s_socket_write_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "__nw_parameters_configure_protocol_disable", referenced from:
        _s_setup_socket_params in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_setup_socket_params in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_cancel", referenced from:
        _s_process_socket_cancel_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_copy_current_path", referenced from:
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_copy_endpoint", referenced from:
        _s_process_listener_success_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_copy_protocol_metadata", referenced from:
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_create", referenced from:
        _s_socket_connect_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_receive", referenced from:
        _s_schedule_next_read in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_send", referenced from:
        _s_socket_write_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_set_queue", referenced from:
        _s_client_set_dispatch_queue in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_set_state_changed_handler", referenced from:
        _s_socket_connect_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_listener_success_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_connection_start", referenced from:
        _s_socket_connect_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_socket_assign_to_event_loop_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_content_context_get_is_final", referenced from:
        ___s_schedule_next_read_block_invoke in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_endpoint_create_address", referenced from:
        _s_socket_connect_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_socket_bind_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_endpoint_get_hostname", referenced from:
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_listener_success_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_endpoint_get_port", referenced from:
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_listener_success_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_error_get_error_code", referenced from:
        _s_handle_connection_state_changed_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        ___s_socket_start_accept_fn_block_invoke in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        ___s_schedule_next_read_block_invoke in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        ___s_socket_write_fn_block_invoke in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_cancel", referenced from:
        _s_socket_stop_accept_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_socket_cancel_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_create", referenced from:
        _s_socket_listen_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_get_port", referenced from:
        _s_process_listener_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_set_new_connection_handler", referenced from:
        _s_socket_start_accept_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_set_queue", referenced from:
        _s_listener_set_dispatch_queue in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_set_state_changed_handler", referenced from:
        _s_socket_start_accept_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_listener_start", referenced from:
        _s_socket_start_accept_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_parameters_create_secure_tcp", referenced from:
        _s_setup_socket_params in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_parameters_create_secure_udp", referenced from:
        _s_setup_socket_params in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_parameters_set_local_endpoint", referenced from:
        _s_socket_bind_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_parameters_set_reuse_local_address", referenced from:
        _s_setup_socket_params in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_path_copy_effective_local_endpoint", referenced from:
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_protocol_copy_tls_definition", referenced from:
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_release", referenced from:
        _s_socket_impl_destroy in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_socket_connect_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_socket_bind_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_setup_socket_params in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_process_connection_state_changed_task in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        ...
    "_nw_retain", referenced from:
        _s_socket_connect_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        _s_socket_listen_fn in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
        ___s_socket_start_accept_fn_block_invoke_2 in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tcp_options_set_connection_timeout", referenced from:
        _s_setup_tcp_options in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tcp_options_set_enable_keepalive", referenced from:
        _s_setup_tcp_options in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tcp_options_set_keepalive_count", referenced from:
        _s_setup_tcp_options in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tcp_options_set_keepalive_idle_time", referenced from:
        _s_setup_tcp_options in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tcp_options_set_keepalive_interval", referenced from:
        _s_setup_tcp_options in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tcp_options_set_maximum_segment_size", referenced from:
        _s_setup_tcp_options in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
    "_nw_tls_copy_sec_protocol_options", referenced from:
        ___s_setup_socket_params_block_invoke in libarrow_bundled_dependencies.a[417](nw_socket.c.o)
  ld: symbol(s) not found for architecture arm64

It seems that we need the Network framework too.

@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 9, 2025
@kou
Copy link
Member Author

kou commented Oct 9, 2025

https://github.com/kou/arrow-java/actions/runs/18334827827/job/52321140360#step:12:2475

   FAILED: [code=1] dataset/libarrow_dataset_jni.dylib 
  : && /usr/bin/c++ -O3 -DNDEBUG -arch arm64 -mmacosx-version-min=14.0 -dynamiclib -Wl,-headerpad_max_install_names -o dataset/libarrow_dataset_jni.dylib -install_name @rpath/libarrow_dataset_jni.dylib dataset/CMakeFiles/arrow_java_jni_dataset.dir/src/main/cpp/jni_wrapper.cc.o dataset/CMakeFiles/arrow_java_jni_dataset.dir/src/main/cpp/jni_util.cc.o  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_dataset.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_substrait.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_dataset.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libparquet.a  /opt/homebrew/Cellar/thrift/0.22.0/lib/libthrift.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_acero.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_compute.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow.a  /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_bundled_dependencies.a  -framework CoreFoundation  -framework Network  -framework Security  /opt/homebrew/opt/openssl@3/lib/libssl.a  /opt/homebrew/opt/openssl@3/lib/libcrypto.a  /opt/homebrew/opt/lz4/lib/liblz4.a  /opt/homebrew/lib/libsnappy.a  /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libz.tbd  /opt/homebrew/lib/libzstd.a  /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libcurl.tbd  /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libcurl.tbd  /opt/homebrew/lib/libutf8proc.a && :
  ld: warning: ignoring duplicate libraries: '/Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_dataset.a'
  duplicate symbol '__ZN5arrow7compute8internal10LocateZoneENSt3__117basic_string_viewIcNS2_11char_traitsIcEEEE' in:
      /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow.a[10](unity_2_cxx.cxx.o)
      /Users/runner/work/arrow-java/arrow-java/build/cpp-install/lib/libarrow_compute.a[4](unity_3_cxx.cxx.o)
  ld: 1 duplicate symbols
  clang: error: linker command failed with exit code 1 (use -v to see invocation)

arrow::compute::internal::LocateZone() is duplicated.

@kou kou added the CI: Extra: C++ Run extra C++ CI label Oct 9, 2025
@kou kou removed the CI: Extra: C++ Run extra C++ CI label Oct 9, 2025
@kou kou changed the title GH-47748: [C++] Fix bundled aws-c-common detection GH-47748: [C++][Dataset] Fix link error on macOS Oct 9, 2025
@kou
Copy link
Member Author

kou commented Oct 9, 2025

Fixed: https://github.com/kou/arrow-java/actions/runs/18370527401/job/52334724160#step:12:2489

  + cmake --build /Users/runner/work/arrow-java/arrow-java/build/java_jni --target install
  [0/1] Install the project...
  -- Install configuration: "release"
  -- Installing: /Users/runner/work/arrow-java/arrow-java/build/java_jni/java-jni/lib/arrow_cdata_jni/aarch_64/libarrow_cdata_jni.dylib
  -- Installing: /Users/runner/work/arrow-java/arrow-java/build/java_jni/java-jni/lib/arrow_dataset_jni/aarch_64/libarrow_dataset_jni.dylib
  -- Installing: /Users/runner/work/arrow-java/arrow-java/build/java_jni/java-jni/lib/gandiva_jni/aarch_64/libgandiva_jni.dylib
  -- Installing: /Users/runner/work/arrow-java/arrow-java/build/java_jni/java-jni/lib/arrow_orc_jni/aarch_64/libarrow_orc_jni.dylib

This is ready.

This error is another problem that should be fixed in apache/arrow-java:

https://github.com/kou/arrow-java/actions/runs/18370527401/job/52334724160#step:12:2535

  Error: Unexpected shared dependency found in arrow_dataset_jni/aarch_64/libarrow_dataset_jni.dylib: `Network`

compute/kernels/scalar_temporal_binary.cc
compute/kernels/scalar_temporal_unary.cc
compute/kernels/scalar_validity.cc
compute/kernels/temporal_internal.cc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a good find! Thanks for the fix!
Does this kernel has to be part of libarrow core, can't it be part of libarrow_compute instead? Maybe @rok
We should try to keep on libarrow only the core kernels (like cast, etcetera) that are required for libarrow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we need this in libarrow.so because

#include "arrow/compute/kernels/temporal_internal.h"
uses this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, let's merge this and maybe we could explore whether this is "really necessary" as a core kernel on a separate issue? @rok ?

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 10, 2025
@raulcd raulcd merged commit 7700dd4 into apache:main Oct 10, 2025
48 checks passed
@raulcd raulcd removed the awaiting changes Awaiting changes label Oct 10, 2025
@github-actions github-actions bot added the awaiting merge Awaiting merge label Oct 10, 2025
raulcd pushed a commit that referenced this pull request Oct 10, 2025
### Rationale for this change

There are link errors with build options for JNI on macOS.

### What changes are included in this PR?

`ARROW_BUNDLED_STATIC_LIBS` has CMake target names defined in Apache Arrow not `find_package()`-ed target names. So we should use `aws-c-common` not `AWS::aws-c-common`.

Recent aws-c-common or something use the Network framework. So add `Network` to `Arrow::arrow_bundled_dependencies` dependencies.

Don't use `compute/kernels/temporal_internal.cc` in `libarrow.dylib` and `libarrow_compute.dylib` to avoid duplicated symbols error.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #47748

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
@kou kou deleted the cpp-arrow-config-macos branch October 10, 2025 12:43
@conbench-apache-arrow
Copy link

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 7700dd4.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

zanmato1984 pushed a commit to zanmato1984/arrow that referenced this pull request Oct 15, 2025
### Rationale for this change

There are link errors with build options for JNI on macOS.

### What changes are included in this PR?

`ARROW_BUNDLED_STATIC_LIBS` has CMake target names defined in Apache Arrow not `find_package()`-ed target names. So we should use `aws-c-common` not `AWS::aws-c-common`.

Recent aws-c-common or something use the Network framework. So add `Network` to `Arrow::arrow_bundled_dependencies` dependencies.

Don't use `compute/kernels/temporal_internal.cc` in `libarrow.dylib` and `libarrow_compute.dylib` to avoid duplicated symbols error.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#47748

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants