Skip to content

[SPARK-51806][BUILD] Upgrade kryo-shaded to 4.0.3 #50586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-51806][BUILD] Upgrade kryo-shaded to 4.0.3 #50586

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
20d93d1
init
LuciferYang Apr 15, 2025
580d755
add comments
LuciferYang Apr 15, 2025
b73e946
comments
LuciferYang Apr 15, 2025
bf2ce6b
update bench result
LuciferYang Apr 15, 2025
fd76d12
exclude
LuciferYang Apr 16, 2025
29cdc32
exclude
LuciferYang Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions common/unsafe/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,14 @@
<groupId>com.twitter</groupId>
<artifactId>chill_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo-shaded</artifactId>
</dependency>
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
</dependency>

<!-- Core dependencies -->
<dependency>
Expand Down
38 changes: 19 additions & 19 deletions core/benchmarks/KryoBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,27 @@
Benchmark Kryo Unsafe vs safe Serialization
================================================================================================

OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
Benchmark Kryo Unsafe vs safe Serialization: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
basicTypes: Int with unsafe:true 166 168 3 6.0 165.5 1.0X
basicTypes: Long with unsafe:true 178 182 5 5.6 178.2 0.9X
basicTypes: Float with unsafe:true 185 189 6 5.4 184.9 0.9X
basicTypes: Double with unsafe:true 183 188 9 5.5 183.2 0.9X
Array: Int with unsafe:true 1 1 0 763.7 1.3 126.4X
Array: Long with unsafe:true 2 2 0 447.3 2.2 74.1X
Array: Float with unsafe:true 1 1 0 753.7 1.3 124.8X
Array: Double with unsafe:true 2 2 0 457.5 2.2 75.7X
Map of string->Double with unsafe:true 28 28 0 36.2 27.6 6.0X
basicTypes: Int with unsafe:false 203 204 1 4.9 203.1 0.8X
basicTypes: Long with unsafe:false 223 224 1 4.5 222.8 0.7X
basicTypes: Float with unsafe:false 206 207 1 4.9 205.8 0.8X
basicTypes: Double with unsafe:false 204 205 1 4.9 204.1 0.8X
Array: Int with unsafe:false 13 13 0 79.5 12.6 13.2X
Array: Long with unsafe:false 21 22 1 46.6 21.5 7.7X
Array: Float with unsafe:false 13 13 0 78.6 12.7 13.0X
Array: Double with unsafe:false 15 15 0 67.8 14.8 11.2X
Map of string->Double with unsafe:false 28 30 1 35.3 28.3 5.8X
basicTypes: Int with unsafe:true 166 169 6 6.0 165.7 1.0X
basicTypes: Long with unsafe:true 178 182 4 5.6 177.5 0.9X
basicTypes: Float with unsafe:true 183 191 9 5.5 182.7 0.9X
basicTypes: Double with unsafe:true 186 193 7 5.4 186.4 0.9X
Array: Int with unsafe:true 1 1 0 745.8 1.3 123.5X
Array: Long with unsafe:true 2 3 0 451.7 2.2 74.8X
Array: Float with unsafe:true 1 1 0 743.0 1.3 123.1X
Array: Double with unsafe:true 2 2 0 475.6 2.1 78.8X
Map of string->Double with unsafe:true 27 28 1 37.0 27.0 6.1X
basicTypes: Int with unsafe:false 198 199 1 5.1 197.7 0.8X
basicTypes: Long with unsafe:false 220 221 1 4.5 219.8 0.8X
basicTypes: Float with unsafe:false 206 208 1 4.8 206.3 0.8X
basicTypes: Double with unsafe:false 222 225 2 4.5 221.9 0.7X
Array: Int with unsafe:false 13 14 1 78.0 12.8 12.9X
Array: Long with unsafe:false 21 21 1 48.2 20.8 8.0X
Array: Float with unsafe:false 6 6 0 178.9 5.6 29.6X
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double in throughput 🤔
Are we are expecting this kind of improvement on a hotfix bump I wonder.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

image

@hiufung-kwok Judging from the historical update records, the current data seems relatively normal, whereas the previous update results may have had some unknown errors. It is also possible that there are certain unstable factors affecting the results of this test case, which could be worth investigating.

Array: Double with unsafe:false 15 16 0 65.3 15.3 10.8X
Map of string->Double with unsafe:false 28 29 2 35.2 28.4 5.8X


38 changes: 19 additions & 19 deletions core/benchmarks/KryoBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,27 @@
Benchmark Kryo Unsafe vs safe Serialization
================================================================================================

OpenJDK 64-Bit Server VM 17.0.14+7-LTS on Linux 6.8.0-1020-azure
OpenJDK 64-Bit Server VM 17.0.14+7-LTS on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
Benchmark Kryo Unsafe vs safe Serialization: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
basicTypes: Int with unsafe:true 177 180 3 5.7 176.9 1.0X
basicTypes: Long with unsafe:true 188 190 1 5.3 188.1 0.9X
basicTypes: Float with unsafe:true 190 192 2 5.3 190.2 0.9X
basicTypes: Double with unsafe:true 199 201 4 5.0 199.0 0.9X
Array: Int with unsafe:true 1 1 0 783.6 1.3 138.6X
Array: Long with unsafe:true 2 2 0 491.6 2.0 87.0X
Array: Float with unsafe:true 1 1 0 757.8 1.3 134.1X
Array: Double with unsafe:true 2 2 0 497.5 2.0 88.0X
Map of string->Double with unsafe:true 26 27 3 37.9 26.4 6.7X
basicTypes: Int with unsafe:false 230 232 1 4.3 230.1 0.8X
basicTypes: Long with unsafe:false 267 268 1 3.7 267.0 0.7X
basicTypes: Float with unsafe:false 229 230 1 4.4 229.2 0.8X
basicTypes: Double with unsafe:false 216 217 1 4.6 216.3 0.8X
Array: Int with unsafe:false 15 15 0 68.8 14.5 12.2X
Array: Long with unsafe:false 22 22 0 46.1 21.7 8.2X
Array: Float with unsafe:false 6 6 0 169.6 5.9 30.0X
Array: Double with unsafe:false 9 9 0 108.1 9.3 19.1X
Map of string->Double with unsafe:false 28 28 1 36.3 27.5 6.4X
basicTypes: Int with unsafe:true 172 174 1 5.8 171.9 1.0X
basicTypes: Long with unsafe:true 196 197 0 5.1 196.2 0.9X
basicTypes: Float with unsafe:true 193 195 2 5.2 192.7 0.9X
basicTypes: Double with unsafe:true 193 194 1 5.2 193.2 0.9X
Array: Int with unsafe:true 1 1 0 715.2 1.4 122.9X
Array: Long with unsafe:true 2 2 0 474.2 2.1 81.5X
Array: Float with unsafe:true 1 1 0 718.2 1.4 123.5X
Array: Double with unsafe:true 2 2 0 475.8 2.1 81.8X
Map of string->Double with unsafe:true 27 28 0 36.7 27.2 6.3X
basicTypes: Int with unsafe:false 207 209 5 4.8 207.3 0.8X
basicTypes: Long with unsafe:false 239 241 2 4.2 238.9 0.7X
basicTypes: Float with unsafe:false 215 217 2 4.6 215.4 0.8X
basicTypes: Double with unsafe:false 220 225 7 4.5 220.2 0.8X
Array: Int with unsafe:false 16 20 7 63.4 15.8 10.9X
Array: Long with unsafe:false 22 22 0 45.9 21.8 7.9X
Array: Float with unsafe:false 6 6 1 170.0 5.9 29.2X
Array: Double with unsafe:false 10 10 0 98.6 10.1 16.9X
Map of string->Double with unsafe:false 28 29 1 35.9 27.9 6.2X


38 changes: 19 additions & 19 deletions core/benchmarks/KryoIteratorBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,27 @@
Benchmark of kryo asIterator on deserialization stream
================================================================================================

OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1021-azure
AMD EPYC 7763 64-Core Processor
Benchmark of kryo asIterator on deserialization stream: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------------
Colletion of int with 1 elements, useIterator: true 6 6 0 1.7 584.3 1.0X
Colletion of int with 10 elements, useIterator: true 13 14 0 0.8 1330.3 0.4X
Colletion of int with 100 elements, useIterator: true 83 84 0 0.1 8310.8 0.1X
Colletion of string with 1 elements, useIterator: true 8 8 0 1.3 763.6 0.8X
Colletion of string with 10 elements, useIterator: true 22 23 0 0.5 2209.8 0.3X
Colletion of string with 100 elements, useIterator: true 163 164 2 0.1 16262.3 0.0X
Colletion of Array[int] with 1 elements, useIterator: true 7 8 0 1.4 730.9 0.8X
Colletion of Array[int] with 10 elements, useIterator: true 20 20 0 0.5 1990.1 0.3X
Colletion of Array[int] with 100 elements, useIterator: true 155 156 1 0.1 15527.8 0.0X
Colletion of int with 1 elements, useIterator: false 6 6 0 1.7 599.4 1.0X
Colletion of int with 10 elements, useIterator: false 13 14 0 0.7 1337.3 0.4X
Colletion of int with 100 elements, useIterator: false 83 84 1 0.1 8320.9 0.1X
Colletion of string with 1 elements, useIterator: false 7 8 0 1.4 731.4 0.8X
Colletion of string with 10 elements, useIterator: false 22 22 0 0.5 2160.9 0.3X
Colletion of string with 100 elements, useIterator: false 170 171 0 0.1 17015.3 0.0X
Colletion of Array[int] with 1 elements, useIterator: false 7 8 1 1.4 710.4 0.8X
Colletion of Array[int] with 10 elements, useIterator: false 19 20 0 0.5 1925.0 0.3X
Colletion of Array[int] with 100 elements, useIterator: false 143 144 2 0.1 14267.3 0.0X
Colletion of int with 1 elements, useIterator: true 6 7 0 1.5 645.9 1.0X
Colletion of int with 10 elements, useIterator: true 13 14 0 0.8 1330.8 0.5X
Colletion of int with 100 elements, useIterator: true 80 81 1 0.1 7987.1 0.1X
Colletion of string with 1 elements, useIterator: true 8 8 0 1.3 787.4 0.8X
Colletion of string with 10 elements, useIterator: true 21 21 0 0.5 2113.7 0.3X
Colletion of string with 100 elements, useIterator: true 161 162 1 0.1 16108.9 0.0X
Colletion of Array[int] with 1 elements, useIterator: true 7 8 0 1.3 747.8 0.9X
Colletion of Array[int] with 10 elements, useIterator: true 19 19 0 0.5 1879.8 0.3X
Colletion of Array[int] with 100 elements, useIterator: true 140 141 1 0.1 14008.3 0.0X
Colletion of int with 1 elements, useIterator: false 6 7 0 1.6 642.6 1.0X
Colletion of int with 10 elements, useIterator: false 14 15 1 0.7 1414.6 0.5X
Colletion of int with 100 elements, useIterator: false 87 88 1 0.1 8699.0 0.1X
Colletion of string with 1 elements, useIterator: false 7 8 0 1.3 746.5 0.9X
Colletion of string with 10 elements, useIterator: false 22 22 0 0.5 2192.3 0.3X
Colletion of string with 100 elements, useIterator: false 161 162 2 0.1 16091.2 0.0X
Colletion of Array[int] with 1 elements, useIterator: false 7 8 0 1.4 719.6 0.9X
Colletion of Array[int] with 10 elements, useIterator: false 19 19 0 0.5 1869.6 0.3X
Colletion of Array[int] with 100 elements, useIterator: false 138 139 1 0.1 13766.1 0.0X


Loading