Skip to content

[CELEBORN-2239] Support Spark 4.1 #3571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SteNicholas wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[CELEBORN-2239] Support Spark 4.1 #3571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/deps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ jobs:
matrix:
module:
- 'spark-4.0'
- 'spark-4.1'
steps:
- uses: actions/checkout@v4
- name: Setup JDK 17
Expand Down
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Celeborn worker's slot count decreases when a partition is allocated and increme

Build Celeborn via `make-distribution.sh`:
```shell
./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pspark-4.0/-Pflink-1.16/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pflink-2.0/-Pflink-2.1/-Pflink-2.2/-Pmr
./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pspark-4.0/-Pspark-4.1/-Pflink-1.16/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pflink-2.0/-Pflink-2.1/-Pflink-2.2/-Pmr
```

Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
Expand All @@ -63,6 +63,7 @@ Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
| Spark 3.4 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Spark 3.5 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Spark 4.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✔ |
| Spark 4.1 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✔ |
| Flink 1.16 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.17 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.18 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
Expand Down
2 changes: 2 additions & 0 deletions build/make-distribution.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,7 @@ if [ "$SBT_ENABLED" == "true" ]; then
sbt_build_client -Pspark-3.5
export JAVA_HOME=$JAVA17_HOME
sbt_build_client -Pspark-4.0
sbt_build_client -Pspark-4.1
export JAVA_HOME=$JAVA8_HOME
sbt_build_client -Pflink-1.16
sbt_build_client -Pflink-1.17
Expand Down Expand Up @@ -399,6 +400,7 @@ else
build_spark_client -Pspark-3.5
export JAVA_HOME=$JAVA17_HOME
build_spark_client -Pspark-4.0
build_spark_client -Pspark-4.1
export JAVA_HOME=$JAVA8_HOME
build_flink_client -Pflink-1.16
build_flink_client -Pflink-1.17
Expand Down
2 changes: 1 addition & 1 deletion build/release/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ upload_nexus_staging() {

export JAVA_HOME=$JAVA17_HOME
echo "Deploying celeborn-client-spark-4-shaded_2.13"
${PROJECT_DIR}/build/sbt -Pspark-4.0 "clean;celeborn-client-spark-4-shaded/publishSigned"
${PROJECT_DIR}/build/sbt -Pspark-4.1 "clean;celeborn-client-spark-4-shaded/publishSigned"
export JAVA_HOME=$JAVA8_HOME

echo "Deploying celeborn-client-flink-1.16-shaded_2.12"
Expand Down
105 changes: 105 additions & 0 deletions dev/deps/dependencies-client-spark-4.1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

RoaringBitmap/1.0.6//RoaringBitmap-1.0.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-io/2.17.0//commons-io-2.17.0.jar
commons-lang3/3.17.0//commons-lang3-3.17.0.jar
commons-logging/1.1.3//commons-logging-1.1.3.jar
failureaccess/1.0.2//failureaccess-1.0.2.jar
guava/33.1.0-jre//guava-33.1.0-jre.jar
hadoop-client-api/3.3.6//hadoop-client-api-3.3.6.jar
hadoop-client-runtime/3.3.6//hadoop-client-runtime-3.3.6.jar
jackson-annotations/2.15.3//jackson-annotations-2.15.3.jar
jackson-core/2.15.3//jackson-core-2.15.3.jar
jackson-databind/2.15.3//jackson-databind-2.15.3.jar
jackson-module-scala_2.13/2.15.3//jackson-module-scala_2.13-2.15.3.jar
jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
jsr305/1.3.9//jsr305-1.3.9.jar
jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
lz4-java/1.8.0//lz4-java-1.8.0.jar
metrics-core/4.2.25//metrics-core-4.2.25.jar
metrics-graphite/4.2.25//metrics-graphite-4.2.25.jar
metrics-jvm/4.2.25//metrics-jvm-4.2.25.jar
netty-all/4.1.118.Final//netty-all-4.1.118.Final.jar
netty-buffer/4.1.118.Final//netty-buffer-4.1.118.Final.jar
netty-buffer/4.2.7.Final//netty-buffer-4.2.7.Final.jar
Copy link
Member

@pan3793 pan3793 Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two versions of netty co-exist does not make sense

Copy link
Member Author

@SteNicholas SteNicholas Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pan3793, I have excluded netty dependencies of spark-sql, which produces 4.2.7.Final version netty dependencies:

[INFO] org.apache.celeborn:celeborn-client-spark-3_2.13:jar:0.7.0-SNAPSHOT
[INFO] +- org.apache.celeborn:celeborn-common_2.13:jar:0.7.0-SNAPSHOT:compile
[INFO] |  +- org.apache.celeborn:celeborn-spi:jar:0.7.0-SNAPSHOT:compile
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:4.2.25:compile
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:4.2.25:compile
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:4.2.25:compile
[INFO] |  +- org.yaml:snakeyaml:jar:2.2:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.10.5:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.36:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.36:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.36:compile
[INFO] |  +- commons-io:commons-io:jar:2.17.0:compile
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.17.0:compile
[INFO] |  +- io.netty:netty-all:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-haproxy:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-memcache:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-mqtt:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-redis:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-smtp:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-stomp:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-codec-xml:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-transport-rxtx:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-transport-sctp:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-transport-udt:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.118.Final:compile
[INFO] |  |  +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.118.Final:runtime
[INFO] |  |  \- io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.118.Final:runtime
[INFO] |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- com.google.guava:guava:jar:33.1.0-jre:compile
[INFO] |  |  \- com.google.guava:failureaccess:jar:1.0.2:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.25.5:compile
[INFO] |  +- org.scala-lang:scala-library:jar:2.13.17:compile
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.13.17:compile
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:1.0.6:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.15.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.3:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.15.3:compile
[INFO] +- org.apache.celeborn:celeborn-client_2.13:jar:0.7.0-SNAPSHOT:compile
[INFO] |  +- org.lz4:lz4-java:jar:1.8.0:compile
[INFO] |  \- com.github.luben:zstd-jni:jar:1.5.7-6:compile
[INFO] +- org.apache.celeborn:celeborn-client-spark-common_2.13:jar:0.7.0-SNAPSHOT:compile
[INFO] +- org.apache.spark:spark-core_2.13:jar:4.1.0:provided
[INFO] |  +- org.scala-lang.modules:scala-parallel-collections_2.13:jar:1.2.0:provided
[INFO] |  +- org.apache.avro:avro:jar:1.12.1:provided
[INFO] |  +- org.apache.avro:avro-mapred:jar:1.12.1:provided
[INFO] |  +- com.twitter:chill_2.13:jar:0.10.0:provided
[INFO] |  +- com.twitter:chill-java:jar:0.10.0:provided
[INFO] |  +- com.esotericsoftware:kryo-shaded:jar:4.0.3:provided
[INFO] |  +- org.objenesis:objenesis:jar:3.4:provided
[INFO] |  +- org.apache.spark:spark-launcher_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-kvstore_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-common_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-unsafe_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-common-utils_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.curator:curator-recipes:jar:5.9.0:provided
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.9.4:provided
[INFO] |  +- io.jsonwebtoken:jjwt-api:jar:0.12.6:provided
[INFO] |  +- io.jsonwebtoken:jjwt-impl:jar:0.12.6:provided
[INFO] |  +- io.jsonwebtoken:jjwt-jackson:jar:0.12.6:provided
[INFO] |  +- commons-codec:commons-codec:jar:1.19.0:provided
[INFO] |  +- org.apache.commons:commons-compress:jar:1.28.0:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.6.1:provided
[INFO] |  +- org.apache.commons:commons-text:jar:1.14.0:provided
[INFO] |  +- org.apache.commons:commons-collections4:jar:4.5.0:provided
[INFO] |  +- com.ning:compress-lzf:jar:1.1.2:provided
[INFO] |  +- org.scala-lang.modules:scala-xml_2.13:jar:2.4.0:provided
[INFO] |  +- org.json4s:json4s-jackson_2.13:jar:4.0.7:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:3.0.18:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:3.0.18:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.47:provided
[INFO] |  |  \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:3.0.18:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.47:provided
[INFO] |  +- org.glassfish.jersey.inject:jersey-hk2:jar:2.47:provided
[INFO] |  +- com.clearspring.analytics:stream:jar:2.9.8:provided
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:4.2.37:provided
[INFO] |  +- io.dropwizard.metrics:metrics-jmx:jar:4.2.37:provided
[INFO] |  +- net.razorvine:pickle:jar:1.5:provided
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.9.9:provided
[INFO] |  \- org.apache.spark:spark-tags_2.13:jar:4.1.0:provided
[INFO] +- org.apache.spark:spark-sql_2.13:jar:4.1.0:provided
[INFO] |  +- org.rocksdb:rocksdbjni:jar:9.10.0:provided
[INFO] |  +- com.univocity:univocity-parsers:jar:2.9.1:provided
[INFO] |  +- org.apache.spark:spark-sketch_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-common-utils-java_2.13:jar:4.1.0:provided
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.8.3:compile
[INFO] |  +- org.apache.ivy:ivy:jar:2.5.3:provided
[INFO] |  +- oro:oro:jar:2.0.8:provided
[INFO] |  +- org.apache.logging.log4j:log4j-slf4j2-impl:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-api:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-core:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-1.2-api:jar:2.24.3:provided
[INFO] |  +- org.apache.logging.log4j:log4j-layout-template-json:jar:2.24.3:provided
[INFO] |  +- org.apache.avro:avro-ipc:jar:1.12.1:provided
[INFO] |  +- org.tukaani:xz:jar:1.10:provided
[INFO] |  +- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-classes:jar:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.74.Final:provided
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.74.Final:provided
[INFO] |  +- com.google.crypto.tink:tink:jar:1.16.0:provided
[INFO] |  +- com.google.code.gson:gson:jar:2.11.0:provided
[INFO] |  +- org.apache.curator:curator-framework:jar:5.9.0:provided
[INFO] |  +- org.apache.curator:curator-client:jar:5.9.0:provided
[INFO] |  +- org.apache.zookeeper:zookeeper-jute:jar:3.9.4:provided
[INFO] |  +- org.apache.yetus:audience-annotations:jar:0.12.0:provided
[INFO] |  +- org.json4s:json4s-core_2.13:jar:4.0.7:provided
[INFO] |  +- org.json4s:json4s-ast_2.13:jar:4.0.7:provided
[INFO] |  +- org.json4s:json4s-scalap_2.13:jar:4.0.7:provided
[INFO] |  +- org.json4s:json4s-jackson-core_2.13:jar:4.0.7:provided
[INFO] |  +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:provided
[INFO] |  +- jakarta.inject:jakarta.inject-api:jar:2.0.1:provided
[INFO] |  +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
[INFO] |  +- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
[INFO] |  +- jakarta.validation:jakarta.validation-api:jar:3.0.2:provided
[INFO] |  +- org.glassfish.hk2:hk2-locator:jar:3.0.6:provided
[INFO] |  +- org.glassfish.hk2.external:aopalliance-repackaged:jar:3.0.6:provided
[INFO] |  +- org.glassfish.hk2:hk2-api:jar:3.0.6:provided
[INFO] |  +- org.glassfish.hk2:hk2-utils:jar:3.0.6:provided
[INFO] |  +- org.javassist:javassist:jar:3.30.2-GA:provided
[INFO] |  +- io.netty:netty-buffer:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-base:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-dns:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-compression:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-http:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-http3:jar:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-unix-common:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-socks:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-protobuf:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-codec-marshalling:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-common:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-handler:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-handler-proxy:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-resolver:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-resolver-dns:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-epoll:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-kqueue:jar:4.2.7.Final:compile
[INFO] |  +- io.netty:netty-transport-classes-io_uring:jar:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-classes-quic:jar:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-riscv64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-io_uring:jar:linux-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-io_uring:jar:linux-aarch_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-io_uring:jar:linux-riscv64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:linux-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:linux-aarch_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:osx-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:osx-aarch_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-codec-native-quic:jar:windows-x86_64:4.2.7.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.2.7.Final:runtime
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.2.7.Final:runtime
[INFO] |  +- org.apache.spark:spark-catalyst_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-sql-api_2.13:jar:4.1.0:provided
[INFO] |  +- org.apache.spark:spark-variant_2.13:jar:4.1.0:provided
[INFO] |  +- org.codehaus.janino:janino:jar:3.1.9:provided
[INFO] |  +- org.codehaus.janino:commons-compiler:jar:3.1.9:provided
[INFO] |  +- org.apache.datasketches:datasketches-java:jar:6.2.0:provided
[INFO] |  +- org.apache.datasketches:datasketches-memory:jar:3.0.2:provided
[INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.13:jar:2.4.0:provided
[INFO] |  +- org.antlr:antlr4-runtime:jar:4.13.1:provided
[INFO] |  +- org.apache.arrow:arrow-vector:jar:18.3.0:provided
[INFO] |  +- org.apache.arrow:arrow-format:jar:18.3.0:provided
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.3:provided
[INFO] |  +- com.google.flatbuffers:flatbuffers-java:jar:25.2.10:provided
[INFO] |  +- org.apache.arrow:arrow-memory-netty:jar:18.3.0:provided
[INFO] |  +- org.apache.arrow:arrow-memory-netty-buffer-patch:jar:18.3.0:provided
[INFO] |  +- org.apache.orc:orc-format:jar:shaded-protobuf:1.1.1:provided
[INFO] |  +- org.apache.orc:orc-core:jar:shaded-protobuf:2.2.1:provided
[INFO] |  +- org.apache.orc:orc-shims:jar:2.2.1:provided
[INFO] |  +- io.airlift:aircompressor:jar:2.0.2:provided
[INFO] |  +- org.threeten:threeten-extra:jar:1.8.0:provided
[INFO] |  +- org.locationtech.jts:jts-core:jar:1.20.0:provided
[INFO] |  +- org.apache.orc:orc-mapreduce:jar:shaded-protobuf:2.2.1:provided
[INFO] |  +- org.apache.hive:hive-storage-api:jar:2.8.1:provided
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-common:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-encoding:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-format-structures:jar:1.16.0:provided
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:provided
[INFO] |  +- org.apache.parquet:parquet-hadoop:jar:1.16.0:provided
[INFO] |  +- org.apache.parquet:parquet-jackson:jar:1.16.0:provided
[INFO] |  +- org.apache.ws.xmlschema:xmlschema-core:jar:2.3.1:provided
[INFO] |  +- org.apache.xbean:xbean-asm9-shaded:jar:4.28:provided
[INFO] |  +- com.ibm.icu:icu4j:jar:77.1:provided
[INFO] |  +- org.apache.arrow:arrow-compression:jar:18.3.0:provided
[INFO] |  \- org.apache.arrow:arrow-memory-core:jar:18.3.0:provided
[INFO] +- org.apache.celeborn:celeborn-client_2.13:test-jar:tests:0.7.0-SNAPSHOT:test
[INFO] +- org.apache.spark:spark-core_2.13:test-jar:tests:4.1.0:test
[INFO] +- javax.servlet:javax.servlet-api:jar:4.0.1:test
[INFO] +- jakarta.servlet:jakarta.servlet-api:jar:5.0.0:test
[INFO] +- org.mockito:mockito-core:jar:4.11.0:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.12.19:test
[INFO] |  \- net.bytebuddy:byte-buddy-agent:jar:1.12.19:test
[INFO] +- org.mockito:mockito-inline:jar:4.11.0:test
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.scalatest:scalatest_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-core_2.13:jar:3.2.16:test
[INFO] |  |  +- org.scalatest:scalatest-compatible:jar:3.2.16:test
[INFO] |  |  \- org.scalactic:scalactic_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-featurespec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-flatspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-freespec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-funsuite_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-funspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-propspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-refspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-wordspec_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-diagrams_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-matchers-core_2.13:jar:3.2.16:test
[INFO] |  +- org.scalatest:scalatest-shouldmatchers_2.13:jar:3.2.16:test
[INFO] |  \- org.scalatest:scalatest-mustmatchers_2.13:jar:3.2.16:test
[INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.6:compile
[INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.6:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.3:runtime
[INFO] \- com.github.olivergondza:maven-jdk-tools-wrapper:jar:0.1:compile
[INFO]    \- jdk.tools:jdk.tools:jar:1.6:system

Copy link
Member

@pan3793 pan3793 Dec 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SteNicholas don't harry to mark unfinished conversations as resolved.

I didn't find where your claimed change is, I also don't expect this work.

I have excluded netty dependencies of spark-sql

netty-codec-base/4.2.7.Final//netty-codec-base-4.2.7.Final.jar
netty-codec-compression/4.2.7.Final//netty-codec-compression-4.2.7.Final.jar
netty-codec-dns/4.1.118.Final//netty-codec-dns-4.1.118.Final.jar
netty-codec-dns/4.2.7.Final//netty-codec-dns-4.2.7.Final.jar
netty-codec-haproxy/4.1.118.Final//netty-codec-haproxy-4.1.118.Final.jar
netty-codec-http/4.1.118.Final//netty-codec-http-4.1.118.Final.jar
netty-codec-http/4.2.7.Final//netty-codec-http-4.2.7.Final.jar
netty-codec-http2/4.1.118.Final//netty-codec-http2-4.1.118.Final.jar
netty-codec-http2/4.2.7.Final//netty-codec-http2-4.2.7.Final.jar
netty-codec-marshalling/4.2.7.Final//netty-codec-marshalling-4.2.7.Final.jar
netty-codec-memcache/4.1.118.Final//netty-codec-memcache-4.1.118.Final.jar
netty-codec-mqtt/4.1.118.Final//netty-codec-mqtt-4.1.118.Final.jar
netty-codec-protobuf/4.2.7.Final//netty-codec-protobuf-4.2.7.Final.jar
netty-codec-redis/4.1.118.Final//netty-codec-redis-4.1.118.Final.jar
netty-codec-smtp/4.1.118.Final//netty-codec-smtp-4.1.118.Final.jar
netty-codec-socks/4.1.118.Final//netty-codec-socks-4.1.118.Final.jar
netty-codec-socks/4.2.7.Final//netty-codec-socks-4.2.7.Final.jar
netty-codec-stomp/4.1.118.Final//netty-codec-stomp-4.1.118.Final.jar
netty-codec-xml/4.1.118.Final//netty-codec-xml-4.1.118.Final.jar
netty-codec/4.1.118.Final//netty-codec-4.1.118.Final.jar
netty-codec/4.2.7.Final//netty-codec-4.2.7.Final.jar
netty-common/4.1.118.Final//netty-common-4.1.118.Final.jar
netty-common/4.2.7.Final//netty-common-4.2.7.Final.jar
netty-handler-proxy/4.1.118.Final//netty-handler-proxy-4.1.118.Final.jar
netty-handler-proxy/4.2.7.Final//netty-handler-proxy-4.2.7.Final.jar
netty-handler/4.1.118.Final//netty-handler-4.1.118.Final.jar
netty-handler/4.2.7.Final//netty-handler-4.2.7.Final.jar
netty-resolver-dns-classes-macos/4.1.118.Final//netty-resolver-dns-classes-macos-4.1.118.Final.jar
netty-resolver-dns-native-macos/4.1.118.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.118.Final-osx-aarch_64.jar
netty-resolver-dns-native-macos/4.1.118.Final/osx-x86_64/netty-resolver-dns-native-macos-4.1.118.Final-osx-x86_64.jar
netty-resolver-dns/4.1.118.Final//netty-resolver-dns-4.1.118.Final.jar
netty-resolver-dns/4.2.7.Final//netty-resolver-dns-4.2.7.Final.jar
netty-resolver/4.1.118.Final//netty-resolver-4.1.118.Final.jar
netty-resolver/4.2.7.Final//netty-resolver-4.2.7.Final.jar
netty-transport-classes-epoll/4.1.118.Final//netty-transport-classes-epoll-4.1.118.Final.jar
netty-transport-classes-epoll/4.2.7.Final//netty-transport-classes-epoll-4.2.7.Final.jar
netty-transport-classes-kqueue/4.1.118.Final//netty-transport-classes-kqueue-4.1.118.Final.jar
netty-transport-classes-kqueue/4.2.7.Final//netty-transport-classes-kqueue-4.2.7.Final.jar
netty-transport-native-epoll/4.1.118.Final/linux-aarch_64/netty-transport-native-epoll-4.1.118.Final-linux-aarch_64.jar
netty-transport-native-epoll/4.1.118.Final/linux-riscv64/netty-transport-native-epoll-4.1.118.Final-linux-riscv64.jar
netty-transport-native-epoll/4.1.118.Final/linux-x86_64/netty-transport-native-epoll-4.1.118.Final-linux-x86_64.jar
netty-transport-native-epoll/4.2.7.Final/linux-aarch_64/netty-transport-native-epoll-4.2.7.Final-linux-aarch_64.jar
netty-transport-native-epoll/4.2.7.Final/linux-riscv64/netty-transport-native-epoll-4.2.7.Final-linux-riscv64.jar
netty-transport-native-epoll/4.2.7.Final/linux-x86_64/netty-transport-native-epoll-4.2.7.Final-linux-x86_64.jar
netty-transport-native-kqueue/4.1.118.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.118.Final-osx-aarch_64.jar
netty-transport-native-kqueue/4.1.118.Final/osx-x86_64/netty-transport-native-kqueue-4.1.118.Final-osx-x86_64.jar
netty-transport-native-kqueue/4.2.7.Final/osx-aarch_64/netty-transport-native-kqueue-4.2.7.Final-osx-aarch_64.jar
netty-transport-native-kqueue/4.2.7.Final/osx-x86_64/netty-transport-native-kqueue-4.2.7.Final-osx-x86_64.jar
netty-transport-native-unix-common/4.1.118.Final//netty-transport-native-unix-common-4.1.118.Final.jar
netty-transport-native-unix-common/4.2.7.Final//netty-transport-native-unix-common-4.2.7.Final.jar
netty-transport-rxtx/4.1.118.Final//netty-transport-rxtx-4.1.118.Final.jar
netty-transport-sctp/4.1.118.Final//netty-transport-sctp-4.1.118.Final.jar
netty-transport-udt/4.1.118.Final//netty-transport-udt-4.1.118.Final.jar
netty-transport/4.1.118.Final//netty-transport-4.1.118.Final.jar
netty-transport/4.2.7.Final//netty-transport-4.2.7.Final.jar
paranamer/2.8.3//paranamer-2.8.3.jar
paranamer/2.8//paranamer-2.8.jar
protobuf-java/3.25.5//protobuf-java-3.25.5.jar
scala-library/2.13.17//scala-library-2.13.17.jar
scala-reflect/2.13.17//scala-reflect-2.13.17.jar
slf4j-api/1.7.36//slf4j-api-1.7.36.jar
snakeyaml/2.2//snakeyaml-2.2.jar
snappy-java/1.1.10.5//snappy-java-1.1.10.5.jar
zstd-jni/1.5.7-6//zstd-jni-1.5.7-6.jar
1 change: 1 addition & 0 deletions dev/reformat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ else
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-3.3
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-3.5
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-4.0
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-4.1
${PROJECT_DIR}/build/mvn spotless:apply -Paws
${PROJECT_DIR}/build/mvn spotless:apply -Paliyun
${PROJECT_DIR}/build/mvn spotless:apply -Pmr
Expand Down
1 change: 1 addition & 0 deletions docs/developers/sbt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ The following table indicates the compatibility of Celeborn Spark and Flink clie
| Spark 3.4 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Spark 3.5 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Spark 4.0 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✔ |
| Spark 4.1 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✔ |
| Flink 1.16 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.17 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
| Flink 1.18 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
Expand Down
19 changes: 19 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1581,6 +1581,25 @@
</properties>
</profile>

<profile>
<id>spark-4.1</id>
<modules>
<module>client-spark/common</module>
<module>client-spark/spark-3</module>
<module>client-spark/spark-3-columnar-common</module>
<module>client-spark/spark-4-columnar-shuffle</module>
<module>client-spark/spark-4-shaded</module>
<module>tests/spark-it</module>
</modules>
<properties>
<lz4-java.version>1.8.0</lz4-java.version>
<scala.version>2.13.17</scala.version>
<scala.binary.version>2.13</scala.binary.version>
<spark.version>4.1.0</spark.version>
<zstd-jni.version>1.5.7-6</zstd-jni.version>
</properties>
</profile>

<profile>
<id>jdk-8</id>
<activation>
Expand Down
18 changes: 18 additions & 0 deletions project/CelebornBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -494,6 +494,7 @@ object Utils {
case Some("spark-3.4") => Some(Spark34)
case Some("spark-3.5") => Some(Spark35)
case Some("spark-4.0") => Some(Spark40)
case Some("spark-4.1") => Some(Spark41)
case _ => None
}

Expand Down Expand Up @@ -942,6 +943,23 @@ object Spark40 extends SparkClientProjects {
override val sparkColumnarShuffleVersion: String = "4"
}

object Spark41 extends SparkClientProjects {

val sparkClientProjectPath = "client-spark/spark-3"
val sparkClientProjectName = "celeborn-client-spark-4"
val sparkClientShadedProjectPath = "client-spark/spark-4-shaded"
val sparkClientShadedProjectName = "celeborn-client-spark-4-shaded"

val lz4JavaVersion = "1.8.0"
val sparkProjectScalaVersion = "2.13.17"

val sparkVersion = "4.1.0"
val zstdJniVersion = "1.5.7-6"
val scalaBinaryVersion = "2.13"

override val sparkColumnarShuffleVersion: String = "4"
}

trait SparkClientProjects {

val sparkClientProjectPath: String
Expand Down
18 changes: 18 additions & 0 deletions tests/spark-it/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,5 +328,23 @@
</dependency>
</dependencies>
</profile>
<profile>
<id>spark-4.1</id>
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
</profiles>
</project>
Loading