Skip to content

Commit e73cbd9

Browse files
LuciferYangLuciferYang
committed
[SPARK-51806][BUILD][4.0] Upgrade kryo-shaded to 4.0.3
### What changes were proposed in this pull request? This pr aims to upgrade `kryo-shaded` from 4.0.2 to 4.0.3. ### Why are the changes needed? This version brings some bug fixes and performance improvements related to chunked encoding: - mproved filling InputChunked buffer: EsotericSoftware/kryo#651 - Support skipping input chunks after a buffer underflow EsotericSoftware/kryo#850 - Avoid flush repeatedly when has finished flushing EsotericSoftware/kryo#978 full changes as follows: - EsotericSoftware/kryo@kryo-parent-4.0.2...kryo-parent-4.0.3 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #50603 from LuciferYang/SPARK-51806-4.0. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
1 parent 25bee87 commit e73cbd9

10 files changed

+130
-83
lines changed

‎common/unsafe/pom.xml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,14 @@
8080
<groupId>com.twitter</groupId>
8181
<artifactId>chill_${scala.binary.version}</artifactId>
8282
</dependency>
83+
<dependency>
84+
<groupId>com.esotericsoftware</groupId>
85+
<artifactId>kryo-shaded</artifactId>
86+
</dependency>
87+
<dependency>
88+
<groupId>org.objenesis</groupId>
89+
<artifactId>objenesis</artifactId>
90+
</dependency>
8391

8492
<!-- Core dependencies -->
8593
<dependency>

‎core/benchmarks/KryoBenchmark-jdk21-results.txt

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,27 @@
22
Benchmark Kryo Unsafe vs safe Serialization
33
================================================================================================
44

5-
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
5+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.8.0-1021-azure
66
AMD EPYC 7763 64-Core Processor
77
Benchmark Kryo Unsafe vs safe Serialization: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
88
---------------------------------------------------------------------------------------------------------------------------
9-
basicTypes: Int with unsafe:true 168 173 7 5.9 168.2 1.0X
10-
basicTypes: Long with unsafe:true 186 191 7 5.4 186.3 0.9X
11-
basicTypes: Float with unsafe:true 183 188 6 5.5 183.2 0.9X
12-
basicTypes: Double with unsafe:true 187 194 9 5.4 186.8 0.9X
13-
Array: Int with unsafe:true 1 1 0 767.7 1.3 129.1X
14-
Array: Long with unsafe:true 2 2 0 491.1 2.0 82.6X
15-
Array: Float with unsafe:true 1 1 0 759.5 1.3 127.8X
16-
Array: Double with unsafe:true 2 2 0 496.6 2.0 83.5X
17-
Map of string->Double with unsafe:true 27 27 1 37.7 26.5 6.3X
18-
basicTypes: Int with unsafe:false 231 232 1 4.3 231.3 0.7X
19-
basicTypes: Long with unsafe:false 237 241 5 4.2 237.3 0.7X
20-
basicTypes: Float with unsafe:false 216 218 2 4.6 216.1 0.8X
21-
basicTypes: Double with unsafe:false 233 235 2 4.3 233.2 0.7X
22-
Array: Int with unsafe:false 13 13 0 78.8 12.7 13.3X
23-
Array: Long with unsafe:false 21 22 0 46.7 21.4 7.9X
24-
Array: Float with unsafe:false 8 8 0 125.7 8.0 21.1X
25-
Array: Double with unsafe:false 11 11 0 90.2 11.1 15.2X
26-
Map of string->Double with unsafe:false 28 28 0 35.8 27.9 6.0X
9+
basicTypes: Int with unsafe:true 169 174 8 5.9 169.0 1.0X
10+
basicTypes: Long with unsafe:true 178 185 6 5.6 178.0 0.9X
11+
basicTypes: Float with unsafe:true 188 194 6 5.3 188.5 0.9X
12+
basicTypes: Double with unsafe:true 191 201 10 5.2 190.5 0.9X
13+
Array: Int with unsafe:true 1 1 0 767.3 1.3 129.7X
14+
Array: Long with unsafe:true 2 2 0 502.3 2.0 84.9X
15+
Array: Float with unsafe:true 1 1 0 759.9 1.3 128.4X
16+
Array: Double with unsafe:true 2 2 0 489.7 2.0 82.7X
17+
Map of string->Double with unsafe:true 27 28 1 36.8 27.2 6.2X
18+
basicTypes: Int with unsafe:false 201 202 1 5.0 201.3 0.8X
19+
basicTypes: Long with unsafe:false 218 224 9 4.6 218.0 0.8X
20+
basicTypes: Float with unsafe:false 196 197 1 5.1 196.1 0.9X
21+
basicTypes: Double with unsafe:false 209 210 1 4.8 209.0 0.8X
22+
Array: Int with unsafe:false 13 13 0 79.4 12.6 13.4X
23+
Array: Long with unsafe:false 21 22 1 46.7 21.4 7.9X
24+
Array: Float with unsafe:false 6 6 0 175.4 5.7 29.6X
25+
Array: Double with unsafe:false 11 11 0 89.6 11.2 15.1X
26+
Map of string->Double with unsafe:false 28 28 1 36.3 27.6 6.1X
2727

2828

‎core/benchmarks/KryoBenchmark-results.txt

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,27 @@
22
Benchmark Kryo Unsafe vs safe Serialization
33
================================================================================================
44

5-
OpenJDK 64-Bit Server VM 17.0.14+7-LTS on Linux 6.8.0-1020-azure
5+
OpenJDK 64-Bit Server VM 17.0.15+6-LTS on Linux 6.8.0-1021-azure
66
AMD EPYC 7763 64-Core Processor
77
Benchmark Kryo Unsafe vs safe Serialization: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
88
---------------------------------------------------------------------------------------------------------------------------
9-
basicTypes: Int with unsafe:true 168 169 1 5.9 168.2 1.0X
10-
basicTypes: Long with unsafe:true 194 195 2 5.2 193.6 0.9X
11-
basicTypes: Float with unsafe:true 189 190 1 5.3 189.2 0.9X
12-
basicTypes: Double with unsafe:true 189 190 1 5.3 188.9 0.9X
13-
Array: Int with unsafe:true 1 1 0 761.6 1.3 128.1X
14-
Array: Long with unsafe:true 2 2 0 500.3 2.0 84.2X
15-
Array: Float with unsafe:true 1 1 0 765.4 1.3 128.8X
16-
Array: Double with unsafe:true 2 2 0 492.7 2.0 82.9X
17-
Map of string->Double with unsafe:true 26 27 0 38.0 26.3 6.4X
18-
basicTypes: Int with unsafe:false 203 204 1 4.9 203.0 0.8X
19-
basicTypes: Long with unsafe:false 231 233 2 4.3 230.6 0.7X
20-
basicTypes: Float with unsafe:false 200 202 3 5.0 199.7 0.8X
21-
basicTypes: Double with unsafe:false 208 211 5 4.8 207.8 0.8X
22-
Array: Int with unsafe:false 15 15 0 65.2 15.3 11.0X
23-
Array: Long with unsafe:false 20 21 0 49.0 20.4 8.2X
24-
Array: Float with unsafe:false 6 6 0 170.6 5.9 28.7X
25-
Array: Double with unsafe:false 10 10 0 103.2 9.7 17.4X
26-
Map of string->Double with unsafe:false 28 28 1 35.9 27.9 6.0X
9+
basicTypes: Int with unsafe:true 169 171 2 5.9 168.8 1.0X
10+
basicTypes: Long with unsafe:true 191 196 8 5.2 191.2 0.9X
11+
basicTypes: Float with unsafe:true 189 190 1 5.3 189.0 0.9X
12+
basicTypes: Double with unsafe:true 195 197 2 5.1 194.9 0.9X
13+
Array: Int with unsafe:true 1 1 0 740.2 1.4 124.9X
14+
Array: Long with unsafe:true 2 2 0 491.0 2.0 82.9X
15+
Array: Float with unsafe:true 1 1 0 719.8 1.4 121.5X
16+
Array: Double with unsafe:true 2 2 0 473.8 2.1 80.0X
17+
Map of string->Double with unsafe:true 27 27 1 36.9 27.1 6.2X
18+
basicTypes: Int with unsafe:false 214 215 0 4.7 214.2 0.8X
19+
basicTypes: Long with unsafe:false 234 235 1 4.3 233.8 0.7X
20+
basicTypes: Float with unsafe:false 209 210 1 4.8 209.0 0.8X
21+
basicTypes: Double with unsafe:false 213 217 3 4.7 213.2 0.8X
22+
Array: Int with unsafe:false 14 15 0 69.2 14.4 11.7X
23+
Array: Long with unsafe:false 21 21 1 48.5 20.6 8.2X
24+
Array: Float with unsafe:false 6 6 0 169.0 5.9 28.5X
25+
Array: Double with unsafe:false 10 10 0 103.7 9.6 17.5X
26+
Map of string->Double with unsafe:false 28 29 1 35.5 28.2 6.0X
2727

2828

‎core/benchmarks/KryoIteratorBenchmark-jdk21-results.txt

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,27 @@
22
Benchmark of kryo asIterator on deserialization stream
33
================================================================================================
44

5-
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
5+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.8.0-1021-azure
66
AMD EPYC 7763 64-Core Processor
77
Benchmark of kryo asIterator on deserialization stream: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
88
---------------------------------------------------------------------------------------------------------------------------------------------
9-
Colletion of int with 1 elements, useIterator: true 6 6 0 1.6 607.2 1.0X
10-
Colletion of int with 10 elements, useIterator: true 13 14 0 0.7 1345.2 0.5X
11-
Colletion of int with 100 elements, useIterator: true 82 83 1 0.1 8195.6 0.1X
12-
Colletion of string with 1 elements, useIterator: true 8 8 0 1.3 761.4 0.8X
13-
Colletion of string with 10 elements, useIterator: true 22 22 0 0.5 2201.7 0.3X
14-
Colletion of string with 100 elements, useIterator: true 154 155 1 0.1 15433.3 0.0X
15-
Colletion of Array[int] with 1 elements, useIterator: true 7 7 1 1.4 712.7 0.9X
16-
Colletion of Array[int] with 10 elements, useIterator: true 20 20 0 0.5 1963.0 0.3X
17-
Colletion of Array[int] with 100 elements, useIterator: true 144 145 1 0.1 14395.8 0.0X
18-
Colletion of int with 1 elements, useIterator: false 7 7 0 1.5 654.5 0.9X
19-
Colletion of int with 10 elements, useIterator: false 16 16 0 0.6 1587.0 0.4X
20-
Colletion of int with 100 elements, useIterator: false 106 107 1 0.1 10568.9 0.1X
21-
Colletion of string with 1 elements, useIterator: false 7 8 0 1.4 738.9 0.8X
22-
Colletion of string with 10 elements, useIterator: false 21 22 1 0.5 2119.5 0.3X
23-
Colletion of string with 100 elements, useIterator: false 157 158 1 0.1 15693.2 0.0X
24-
Colletion of Array[int] with 1 elements, useIterator: false 7 7 0 1.4 695.9 0.9X
25-
Colletion of Array[int] with 10 elements, useIterator: false 18 19 1 0.6 1817.1 0.3X
26-
Colletion of Array[int] with 100 elements, useIterator: false 136 137 0 0.1 13612.6 0.0X
9+
Colletion of int with 1 elements, useIterator: true 6 6 0 1.7 601.4 1.0X
10+
Colletion of int with 10 elements, useIterator: true 13 14 0 0.7 1334.9 0.5X
11+
Colletion of int with 100 elements, useIterator: true 83 83 1 0.1 8253.6 0.1X
12+
Colletion of string with 1 elements, useIterator: true 8 8 0 1.3 771.0 0.8X
13+
Colletion of string with 10 elements, useIterator: true 23 23 1 0.4 2256.5 0.3X
14+
Colletion of string with 100 elements, useIterator: true 167 167 0 0.1 16656.6 0.0X
15+
Colletion of Array[int] with 1 elements, useIterator: true 7 8 0 1.4 729.6 0.8X
16+
Colletion of Array[int] with 10 elements, useIterator: true 20 20 0 0.5 1968.9 0.3X
17+
Colletion of Array[int] with 100 elements, useIterator: true 148 149 0 0.1 14841.9 0.0X
18+
Colletion of int with 1 elements, useIterator: false 6 7 0 1.6 621.7 1.0X
19+
Colletion of int with 10 elements, useIterator: false 14 14 0 0.7 1358.1 0.4X
20+
Colletion of int with 100 elements, useIterator: false 86 87 1 0.1 8573.1 0.1X
21+
Colletion of string with 1 elements, useIterator: false 7 7 0 1.4 712.8 0.8X
22+
Colletion of string with 10 elements, useIterator: false 21 21 0 0.5 2109.8 0.3X
23+
Colletion of string with 100 elements, useIterator: false 157 157 0 0.1 15668.4 0.0X
24+
Colletion of Array[int] with 1 elements, useIterator: false 7 7 0 1.4 691.1 0.9X
25+
Colletion of Array[int] with 10 elements, useIterator: false 19 19 0 0.5 1857.8 0.3X
26+
Colletion of Array[int] with 100 elements, useIterator: false 139 140 1 0.1 13894.5 0.0X
2727

2828

0 commit comments

Comments
 (0)