Refactor create_hashes to accept array references #18448

adriangb · 2025-11-02T18:52:15Z

Background

This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in #17171.

A "target state" is tracked in #18393.
There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own:

(This PR): Refactor create_hashes to accept array references #18448
Refactor InListExpr to support structs by re-using existing hashing infrastructure #18449 (depends on Refactor create_hashes to accept array references #18448)
Refactor state management in HashJoinExec and use CASE expressions for more precise filters #18451

Changes in this PR

Change create_hashes and related functions to work with &dyn Array references instead of requiring ArrayRef (Arc-wrapped arrays). This avoids unnecessary Arc::clone() calls and enables calls that only have an &dyn Array to use the hashing utilities.

Add create_hashes_from_arrays(&[&dyn Array]) function
Refactor hash_dictionary, hash_list_array, hash_fixed_list_array to use references instead of cloning
Extract hash_single_array() helper for common logic

2010YOUY01 · 2025-11-03T01:25:32Z

What is this series of PRs implementing? Is it for query anti-pattern in #18393 (comment), stat pruning is not working, so we're pushing down build-side dynamic filter like key in [1,5,23] to probe side parquet scan, and use parquet builtin bloom filter to effectively skip prunable units?

By the way I suggest to open an umbrella issue to describe the high-level ideas, now the discussion seems to be scattered around, and the work is hard to follow.

adriangb · 2025-11-03T01:29:30Z

What is this series of PRs implementing? Is it for query anti-pattern in #18393 (comment), stat pruning is not working, so we're pushing down build-side dynamic filter like key in [1,5,23] to probe side parquet scan, and use parquet builtin bloom filter to effectively skip prunable units?

Yes that is precisely it. I'm not sure what you mean by query anti-pattern though.

I'll put a summary and links to the relevant PRs in #17171 (comment). That said #17171 is already a pretty busy issue so it's buried at the bottom... I'm not sure what more I can do about that.

2010YOUY01 · 2025-11-03T01:46:41Z

What is this series of PRs implementing? Is it for query anti-pattern in #18393 (comment), stat pruning is not working, so we're pushing down build-side dynamic filter like key in [1,5,23] to probe side parquet scan, and use parquet builtin bloom filter to effectively skip prunable units?

Yes that is precisely it. I'm not sure what you mean by query anti-pattern though.

Maybe query anti-pattern is not accurate, but I'm just describing in that query, stat pruning is not working due to the given parquet file is not well clustered on the pruning column.

I'll put a summary and links to the relevant PRs in #17171 (comment). That said #17171 is already a pretty busy issue so it's buried at the bottom... I'm not sure what more I can do about that.

Perhaps edit the issue top description, or open a new one? 🤔

adriangb · 2025-11-03T05:12:05Z

Maybe query anti-pattern is not accurate, but I'm just describing in that query, stat pruning is not working due to the given parquet file is not well clustered on the pruning column.

Okay yes agreed. Anti-pattern was only confusing to me because I thought it might be suggesting that pattern of query is wrong / that the query itself is wrong and had me thinking maybe my example was a bad one 😔

I'll put a summary and links to the relevant PRs in #17171 (comment). That said #17171 is already a pretty busy issue so it's buried at the bottom... I'm not sure what more I can do about that.

Perhaps edit the issue top description, or open a new one? 🤔

Hmm there's others working on the same issue I don't want to make my work sounds like the only solution. I'll add a more extensive description and cross references to the PRs tomorrow.

datafusion/common/src/hash_utils.rs

datafusion/common/src/scalar/mod.rs

alamb

Thanks @adriangb

alamb · 2025-11-07T11:18:28Z

datafusion/common/src/hash_utils.rs

+///
+/// This is the same as [`create_hashes`] but accepts `&dyn Array`s instead of requiring
+/// `ArrayRef`s.
+pub fn create_hashes_from_arrays<'a>(


I think we can avoid this new function and instead update create_hashes with a little generics magic so it can take array references

Here is a PR that shows how it works

Use generics in create_hashes pydantic/datafusion#42

alamb · 2025-11-07T16:18:00Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor-create-hashes (e50907a) to f57da83 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb

I am somewhat biased, but this looks good to me 👍

it looks like this PR has some conflicts that need to be resolved, and I am running some benchmarks on it just to be sure it doesn't impact performance (I don't expect that it would)

alamb · 2025-11-07T16:20:16Z

datafusion/common/src/hash_utils.rs

+    Ok(())
+}
+
+pub trait AsDynArray {


Suggested change

pub trait AsDynArray {

/// Something that can be returned as a `&dyn Array`

///

/// For some reason we can't use `AsRef<dyn Array>` because

/// it is not implemented for`&dyn Array`

pub trait AsDynArray {

Co-authored-by: Andrew Lamb <[email protected]>

alamb · 2025-11-07T17:19:29Z

🤖: Benchmark completed

Details

Comparing HEAD and refactor-create-hashes
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ refactor-create-hashes ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2819.33 ms │             2703.99 ms │     no change │
│ QQuery 1     │  1346.83 ms │             1231.46 ms │ +1.09x faster │
│ QQuery 2     │  2609.07 ms │             2435.67 ms │ +1.07x faster │
│ QQuery 3     │  1211.65 ms │             1186.69 ms │     no change │
│ QQuery 4     │  2299.37 ms │             2274.89 ms │     no change │
│ QQuery 5     │ 28190.03 ms │            28131.02 ms │     no change │
│ QQuery 6     │  4278.94 ms │             4233.17 ms │     no change │
│ QQuery 7     │  3680.04 ms │             3565.94 ms │     no change │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 46435.26ms │
│ Total Time (refactor-create-hashes)   │ 45762.82ms │
│ Average Time (HEAD)                   │  5804.41ms │
│ Average Time (refactor-create-hashes) │  5720.35ms │
│ Queries Faster                        │          2 │
│ Queries Slower                        │          0 │
│ Queries with No Change                │          6 │
│ Queries with Failure                  │          0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ refactor-create-hashes ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.54 ms │                2.59 ms │     no change │
│ QQuery 1     │    53.61 ms │               49.98 ms │ +1.07x faster │
│ QQuery 2     │   138.33 ms │              136.09 ms │     no change │
│ QQuery 3     │   163.42 ms │              167.28 ms │     no change │
│ QQuery 4     │  1085.80 ms │             1098.89 ms │     no change │
│ QQuery 5     │  1519.76 ms │             1508.98 ms │     no change │
│ QQuery 6     │     2.20 ms │                2.35 ms │  1.07x slower │
│ QQuery 7     │    56.16 ms │               55.42 ms │     no change │
│ QQuery 8     │  1496.60 ms │             1482.97 ms │     no change │
│ QQuery 9     │  1871.59 ms │             1861.79 ms │     no change │
│ QQuery 10    │   404.23 ms │              396.73 ms │     no change │
│ QQuery 11    │   460.52 ms │              462.95 ms │     no change │
│ QQuery 12    │  1427.43 ms │             1423.20 ms │     no change │
│ QQuery 13    │  2172.02 ms │             2197.55 ms │     no change │
│ QQuery 14    │  1311.58 ms │             1307.64 ms │     no change │
│ QQuery 15    │  1257.88 ms │             1224.69 ms │     no change │
│ QQuery 16    │  2713.32 ms │             2726.42 ms │     no change │
│ QQuery 17    │  2717.22 ms │             2718.99 ms │     no change │
│ QQuery 18    │  5374.18 ms │             5071.17 ms │ +1.06x faster │
│ QQuery 19    │   127.28 ms │              130.76 ms │     no change │
│ QQuery 20    │  2040.97 ms │             2014.82 ms │     no change │
│ QQuery 21    │  2362.71 ms │             2372.92 ms │     no change │
│ QQuery 22    │  3954.49 ms │             4025.90 ms │     no change │
│ QQuery 23    │ 13021.09 ms │            12987.25 ms │     no change │
│ QQuery 24    │   216.63 ms │              219.51 ms │     no change │
│ QQuery 25    │   505.58 ms │              519.98 ms │     no change │
│ QQuery 26    │   224.50 ms │              228.64 ms │     no change │
│ QQuery 27    │  2912.16 ms │             2954.73 ms │     no change │
│ QQuery 28    │ 24399.79 ms │            22653.71 ms │ +1.08x faster │
│ QQuery 29    │   973.57 ms │              992.15 ms │     no change │
│ QQuery 30    │  1375.06 ms │             1337.68 ms │     no change │
│ QQuery 31    │  1373.01 ms │             1359.86 ms │     no change │
│ QQuery 32    │  5146.36 ms │             4900.38 ms │     no change │
│ QQuery 33    │  6188.54 ms │             6040.24 ms │     no change │
│ QQuery 34    │  6021.22 ms │             6150.16 ms │     no change │
│ QQuery 35    │  2054.38 ms │             2037.85 ms │     no change │
│ QQuery 36    │   121.03 ms │              124.13 ms │     no change │
│ QQuery 37    │    52.00 ms │               54.16 ms │     no change │
│ QQuery 38    │   124.96 ms │              122.84 ms │     no change │
│ QQuery 39    │   199.41 ms │              201.01 ms │     no change │
│ QQuery 40    │    44.38 ms │               41.52 ms │ +1.07x faster │
│ QQuery 41    │    40.22 ms │               38.89 ms │     no change │
│ QQuery 42    │    32.40 ms │               31.84 ms │     no change │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 97740.14ms │
│ Total Time (refactor-create-hashes)   │ 95436.63ms │
│ Average Time (HEAD)                   │  2273.03ms │
│ Average Time (refactor-create-hashes) │  2219.46ms │
│ Queries Faster                        │          4 │
│ Queries Slower                        │          1 │
│ Queries with No Change                │         38 │
│ Queries with Failure                  │          0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ refactor-create-hashes ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 203.43 ms │              169.78 ms │ +1.20x faster │
│ QQuery 2     │  31.37 ms │               26.67 ms │ +1.18x faster │
│ QQuery 3     │  45.59 ms │               37.42 ms │ +1.22x faster │
│ QQuery 4     │  35.40 ms │               28.32 ms │ +1.25x faster │
│ QQuery 5     │  83.54 ms │               77.54 ms │ +1.08x faster │
│ QQuery 6     │  20.03 ms │               19.71 ms │     no change │
│ QQuery 7     │ 226.49 ms │              223.46 ms │     no change │
│ QQuery 8     │  33.46 ms │               33.44 ms │     no change │
│ QQuery 9     │ 101.36 ms │              100.26 ms │     no change │
│ QQuery 10    │  60.46 ms │               59.33 ms │     no change │
│ QQuery 11    │  17.24 ms │               17.31 ms │     no change │
│ QQuery 12    │  51.20 ms │               51.36 ms │     no change │
│ QQuery 13    │  46.45 ms │               47.32 ms │     no change │
│ QQuery 14    │  14.43 ms │               13.73 ms │     no change │
│ QQuery 15    │  25.00 ms │               24.67 ms │     no change │
│ QQuery 16    │  25.30 ms │               24.78 ms │     no change │
│ QQuery 17    │ 150.74 ms │              147.68 ms │     no change │
│ QQuery 18    │ 328.21 ms │              321.42 ms │     no change │
│ QQuery 19    │  36.53 ms │               36.93 ms │     no change │
│ QQuery 20    │  48.61 ms │               49.59 ms │     no change │
│ QQuery 21    │ 313.76 ms │              352.03 ms │  1.12x slower │
│ QQuery 22    │  20.39 ms │               21.15 ms │     no change │
└──────────────┴───────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 1918.99ms │
│ Total Time (refactor-create-hashes)   │ 1883.92ms │
│ Average Time (HEAD)                   │   87.23ms │
│ Average Time (refactor-create-hashes) │   85.63ms │
│ Queries Faster                        │         5 │
│ Queries Slower                        │         1 │
│ Queries with No Change                │        16 │
│ Queries with Failure                  │         0 │
└───────────────────────────────────────┴───────────┘

adriangb · 2025-11-07T17:25:14Z

QQuery 4 │ 35.40 ms │ 28.32 ms │ +1.25x faster

No way this is real 😆

But generally benches look good! Either an improvement or no change + noise. So I will merge this once CI passes. Thank you @alamb and @martin-g !

alamb · 2025-11-07T17:57:13Z

I'll reschedule to double check (no need to wait for merge)

alamb · 2025-11-07T18:46:27Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing refactor-create-hashes (976d2cd) to f32984b diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-11-07T19:44:38Z

🤖: Benchmark completed

Details

Comparing HEAD and refactor-create-hashes
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ refactor-create-hashes ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2689.38 ms │             2698.70 ms │     no change │
│ QQuery 1     │  1266.59 ms │             1174.60 ms │ +1.08x faster │
│ QQuery 2     │  2431.84 ms │             2308.15 ms │ +1.05x faster │
│ QQuery 3     │  1180.08 ms │             1166.20 ms │     no change │
│ QQuery 4     │  2338.77 ms │             2342.68 ms │     no change │
│ QQuery 5     │ 28232.41 ms │            28359.58 ms │     no change │
│ QQuery 6     │  4076.58 ms │             4209.84 ms │     no change │
│ QQuery 7     │  3774.99 ms │             3640.46 ms │     no change │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 45990.64ms │
│ Total Time (refactor-create-hashes)   │ 45900.21ms │
│ Average Time (HEAD)                   │  5748.83ms │
│ Average Time (refactor-create-hashes) │  5737.53ms │
│ Queries Faster                        │          2 │
│ Queries Slower                        │          0 │
│ Queries with No Change                │          6 │
│ Queries with Failure                  │          0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ refactor-create-hashes ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.09 ms │                2.16 ms │     no change │
│ QQuery 1     │    48.82 ms │               49.36 ms │     no change │
│ QQuery 2     │   135.54 ms │              136.99 ms │     no change │
│ QQuery 3     │   161.07 ms │              161.90 ms │     no change │
│ QQuery 4     │  1067.97 ms │             1083.90 ms │     no change │
│ QQuery 5     │  1534.15 ms │             1498.98 ms │     no change │
│ QQuery 6     │     2.25 ms │                2.16 ms │     no change │
│ QQuery 7     │    55.18 ms │               52.41 ms │ +1.05x faster │
│ QQuery 8     │  1465.53 ms │             1504.45 ms │     no change │
│ QQuery 9     │  1841.30 ms │             1845.58 ms │     no change │
│ QQuery 10    │   379.08 ms │              415.64 ms │  1.10x slower │
│ QQuery 11    │   433.01 ms │              430.40 ms │     no change │
│ QQuery 12    │  1361.46 ms │             1380.62 ms │     no change │
│ QQuery 13    │  2161.80 ms │             2172.71 ms │     no change │
│ QQuery 14    │  1274.98 ms │             1286.97 ms │     no change │
│ QQuery 15    │  1262.96 ms │             1294.72 ms │     no change │
│ QQuery 16    │  2679.92 ms │             2719.24 ms │     no change │
│ QQuery 17    │  2669.17 ms │             2716.97 ms │     no change │
│ QQuery 18    │  5423.51 ms │             5101.51 ms │ +1.06x faster │
│ QQuery 19    │   127.86 ms │              126.39 ms │     no change │
│ QQuery 20    │  2052.11 ms │             1997.13 ms │     no change │
│ QQuery 21    │  2335.79 ms │             2308.85 ms │     no change │
│ QQuery 22    │  4031.93 ms │             3970.47 ms │     no change │
│ QQuery 23    │ 18894.65 ms │            12739.39 ms │ +1.48x faster │
│ QQuery 24    │   215.78 ms │              221.49 ms │     no change │
│ QQuery 25    │   472.47 ms │              474.05 ms │     no change │
│ QQuery 26    │   231.23 ms │              223.57 ms │     no change │
│ QQuery 27    │  2869.59 ms │             2801.37 ms │     no change │
│ QQuery 28    │ 24514.24 ms │            23298.88 ms │     no change │
│ QQuery 29    │   968.25 ms │             1005.67 ms │     no change │
│ QQuery 30    │  1377.51 ms │             1353.31 ms │     no change │
│ QQuery 31    │  1400.71 ms │             1387.23 ms │     no change │
│ QQuery 32    │  4947.96 ms │             4891.28 ms │     no change │
│ QQuery 33    │  6123.77 ms │             5958.08 ms │     no change │
│ QQuery 34    │  6316.95 ms │             6037.89 ms │     no change │
│ QQuery 35    │  2116.60 ms │             2080.72 ms │     no change │
│ QQuery 36    │   124.64 ms │              119.78 ms │     no change │
│ QQuery 37    │    53.84 ms │               50.95 ms │ +1.06x faster │
│ QQuery 38    │   121.04 ms │              121.65 ms │     no change │
│ QQuery 39    │   198.78 ms │              196.94 ms │     no change │
│ QQuery 40    │    42.56 ms │               43.73 ms │     no change │
│ QQuery 41    │    40.12 ms │               39.06 ms │     no change │
│ QQuery 42    │    32.79 ms │               32.41 ms │     no change │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 103570.96ms │
│ Total Time (refactor-create-hashes)   │  95336.96ms │
│ Average Time (HEAD)                   │   2408.63ms │
│ Average Time (refactor-create-hashes) │   2217.14ms │
│ Queries Faster                        │           4 │
│ Queries Slower                        │           1 │
│ Queries with No Change                │          38 │
│ Queries with Failure                  │           0 │
└───────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ refactor-create-hashes ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 141.11 ms │              130.63 ms │ +1.08x faster │
│ QQuery 2     │  27.70 ms │               28.69 ms │     no change │
│ QQuery 3     │  38.77 ms │               38.14 ms │     no change │
│ QQuery 4     │  28.88 ms │               28.69 ms │     no change │
│ QQuery 5     │  87.27 ms │               85.02 ms │     no change │
│ QQuery 6     │  19.61 ms │               19.61 ms │     no change │
│ QQuery 7     │ 223.10 ms │              220.34 ms │     no change │
│ QQuery 8     │  38.74 ms │               34.68 ms │ +1.12x faster │
│ QQuery 9     │ 112.03 ms │               99.91 ms │ +1.12x faster │
│ QQuery 10    │  63.74 ms │               62.85 ms │     no change │
│ QQuery 11    │  18.17 ms │               17.09 ms │ +1.06x faster │
│ QQuery 12    │  52.78 ms │               50.13 ms │ +1.05x faster │
│ QQuery 13    │  47.38 ms │               46.39 ms │     no change │
│ QQuery 14    │  13.73 ms │               14.08 ms │     no change │
│ QQuery 15    │  24.81 ms │               24.91 ms │     no change │
│ QQuery 16    │  24.98 ms │               24.85 ms │     no change │
│ QQuery 17    │ 152.79 ms │              148.86 ms │     no change │
│ QQuery 18    │ 281.32 ms │              278.54 ms │     no change │
│ QQuery 19    │  38.06 ms │               38.35 ms │     no change │
│ QQuery 20    │  49.75 ms │               50.24 ms │     no change │
│ QQuery 21    │ 337.78 ms │              325.47 ms │     no change │
│ QQuery 22    │  21.12 ms │               21.31 ms │     no change │
└──────────────┴───────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                     ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                     │ 1843.61ms │
│ Total Time (refactor-create-hashes)   │ 1788.80ms │
│ Average Time (HEAD)                   │   83.80ms │
│ Average Time (refactor-create-hashes) │   81.31ms │
│ Queries Faster                        │         5 │
│ Queries Slower                        │         0 │
│ Queries with No Change                │        17 │
│ Queries with Failure                  │         0 │
└───────────────────────────────────────┴───────────┘

alamb · 2025-11-07T21:09:45Z

it is strange that the second run also shows speedups. I agree that overall it looks good to me and shipping is the right solution here

adriangb · 2025-11-07T21:16:39Z

Not the same speedups / in the same queries though. I think it's just noise. Unfortunate that noise is +-25%...

alamb · 2025-11-08T09:39:27Z

Not the same speedups / in the same queries though. I think it's just noise. Unfortunate that noise is +-25%...

Yeah, I agree

FYI @rluvaton had a great idea here:

Automated way to run benchmarks on a dedicated machine from PRs #18115

## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - (This PR): apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 ## Changes in this PR Change create_hashes and related functions to work with &dyn Array references instead of requiring ArrayRef (Arc-wrapped arrays). This avoids unnecessary Arc::clone() calls and enables calls that only have an &dyn Array to use the hashing utilities. - Add create_hashes_from_arrays(&[&dyn Array]) function - Refactor hash_dictionary, hash_list_array, hash_fixed_list_array to use references instead of cloning - Extract hash_single_array() helper for common logic --------- Co-authored-by: Andrew Lamb <[email protected]>

…nfrastructure (#18449) ## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in #17171. A "target state" is tracked in #18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - #18448 - (This PR): #18449 (depends on #18448) - #18451 ## Changes in this PR - Enhance InListExpr to efficiently store homogeneous lists as arrays and avoid a conversion to Vec<PhysicalExpr> by adding an internal InListStorage enum with Array and Exprs variants - Re-use existing hashing and comparison utilities to support Struct arrays and other complex types - Add public function `in_list_from_array(expr, list_array, negated)` for creating InList from arrays Although the diff looks large most of it is actually tests and docs. I think the actual code change is a negative LOC change, or at least negative complexity (eliminates a trait, a macro, matching on data types). --------- Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

…for more precise filters (#18451) ## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in #17171. A "target state" is tracked in #18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - #18448 - #18449 (depends on #18448) - (This PR): #18451 ## Changes in this PR This PR refactors state management in HashJoinExec to make filter pushdown more efficient and prepare for pushing down membership tests. - Refactor internal data structures to clean up state management and make usage more idiomatic (use `Option` instead of comparing integers, etc.) - Uses CASE expressions to evaluate pushed-down filters selectively by partition Example: `CASE hash_repartition % N WHEN partition_id THEN condition ELSE false END` --------- Co-authored-by: Lía Adriana <[email protected]>

…nfrastructure (apache#18449) ## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - (This PR): apache#18449 (depends on apache#18448) - apache#18451 ## Changes in this PR - Enhance InListExpr to efficiently store homogeneous lists as arrays and avoid a conversion to Vec<PhysicalExpr> by adding an internal InListStorage enum with Array and Exprs variants - Re-use existing hashing and comparison utilities to support Struct arrays and other complex types - Add public function `in_list_from_array(expr, list_array, negated)` for creating InList from arrays Although the diff looks large most of it is actually tests and docs. I think the actual code change is a negative LOC change, or at least negative complexity (eliminates a trait, a macro, matching on data types). --------- Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

…for more precise filters (apache#18451) ## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - (This PR): apache#18451 ## Changes in this PR This PR refactors state management in HashJoinExec to make filter pushdown more efficient and prepare for pushing down membership tests. - Refactor internal data structures to clean up state management and make usage more idiomatic (use `Option` instead of comparing integers, etc.) - Uses CASE expressions to evaluate pushed-down filters selectively by partition Example: `CASE hash_repartition % N WHEN partition_id THEN condition ELSE false END` --------- Co-authored-by: Lía Adriana <[email protected]>

… on the size of the build side (#18393) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in #17171. A "target state" is tracked in #18393 (*this PR*). There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - #18448 - #18449 (depends on #18448) - #18451 As those are merged I will rebase this PR to keep track of the "remaining work", and we can use this PR to explore big picture ideas or benchmarks of the final state.

This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - (This PR): apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 Change create_hashes and related functions to work with &dyn Array references instead of requiring ArrayRef (Arc-wrapped arrays). This avoids unnecessary Arc::clone() calls and enables calls that only have an &dyn Array to use the hashing utilities. - Add create_hashes_from_arrays(&[&dyn Array]) function - Refactor hash_dictionary, hash_list_array, hash_fixed_list_array to use references instead of cloning - Extract hash_single_array() helper for common logic --------- Co-authored-by: Andrew Lamb <[email protected]> (cherry picked from commit a899ca0)

…nfrastructure (apache#18449) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - (This PR): apache#18449 (depends on apache#18448) - apache#18451 - Enhance InListExpr to efficiently store homogeneous lists as arrays and avoid a conversion to Vec<PhysicalExpr> by adding an internal InListStorage enum with Array and Exprs variants - Re-use existing hashing and comparison utilities to support Struct arrays and other complex types - Add public function `in_list_from_array(expr, list_array, negated)` for creating InList from arrays Although the diff looks large most of it is actually tests and docs. I think the actual code change is a negative LOC change, or at least negative complexity (eliminates a trait, a macro, matching on data types). --------- Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> (cherry picked from commit 486c5d8)

…for more precise filters (apache#18451) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - (This PR): apache#18451 This PR refactors state management in HashJoinExec to make filter pushdown more efficient and prepare for pushing down membership tests. - Refactor internal data structures to clean up state management and make usage more idiomatic (use `Option` instead of comparing integers, etc.) - Uses CASE expressions to evaluate pushed-down filters selectively by partition Example: `CASE hash_repartition % N WHEN partition_id THEN condition ELSE false END` --------- Co-authored-by: Lía Adriana <[email protected]> (cherry picked from commit 5b0aa37)

… on the size of the build side (apache#18393) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393 (*this PR*). There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 As those are merged I will rebase this PR to keep track of the "remaining work", and we can use this PR to explore big picture ideas or benchmarks of the final state. (cherry picked from commit c0e8bb5)

This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - (This PR): apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 Change create_hashes and related functions to work with &dyn Array references instead of requiring ArrayRef (Arc-wrapped arrays). This avoids unnecessary Arc::clone() calls and enables calls that only have an &dyn Array to use the hashing utilities. - Add create_hashes_from_arrays(&[&dyn Array]) function - Refactor hash_dictionary, hash_list_array, hash_fixed_list_array to use references instead of cloning - Extract hash_single_array() helper for common logic --------- Co-authored-by: Andrew Lamb <[email protected]> (cherry picked from commit a899ca0) (cherry picked from commit e53debb)

…nfrastructure (apache#18449) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - (This PR): apache#18449 (depends on apache#18448) - apache#18451 - Enhance InListExpr to efficiently store homogeneous lists as arrays and avoid a conversion to Vec<PhysicalExpr> by adding an internal InListStorage enum with Array and Exprs variants - Re-use existing hashing and comparison utilities to support Struct arrays and other complex types - Add public function `in_list_from_array(expr, list_array, negated)` for creating InList from arrays Although the diff looks large most of it is actually tests and docs. I think the actual code change is a negative LOC change, or at least negative complexity (eliminates a trait, a macro, matching on data types). --------- Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> (cherry picked from commit 486c5d8) (cherry picked from commit 181e058)

…for more precise filters (apache#18451) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - (This PR): apache#18451 This PR refactors state management in HashJoinExec to make filter pushdown more efficient and prepare for pushing down membership tests. - Refactor internal data structures to clean up state management and make usage more idiomatic (use `Option` instead of comparing integers, etc.) - Uses CASE expressions to evaluate pushed-down filters selectively by partition Example: `CASE hash_repartition % N WHEN partition_id THEN condition ELSE false END` --------- Co-authored-by: Lía Adriana <[email protected]> (cherry picked from commit 5b0aa37) (cherry picked from commit e9d1985)

… on the size of the build side (apache#18393) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393 (*this PR*). There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 As those are merged I will rebase this PR to keep track of the "remaining work", and we can use this PR to explore big picture ideas or benchmarks of the final state. (cherry picked from commit c0e8bb5) (cherry picked from commit 115313c)

…nfrastructure (apache#18449) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - (This PR): apache#18449 (depends on apache#18448) - apache#18451 - Enhance InListExpr to efficiently store homogeneous lists as arrays and avoid a conversion to Vec<PhysicalExpr> by adding an internal InListStorage enum with Array and Exprs variants - Re-use existing hashing and comparison utilities to support Struct arrays and other complex types - Add public function `in_list_from_array(expr, list_array, negated)` for creating InList from arrays Although the diff looks large most of it is actually tests and docs. I think the actual code change is a negative LOC change, or at least negative complexity (eliminates a trait, a macro, matching on data types). --------- Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> (cherry picked from commit 486c5d8)

…for more precise filters (apache#18451) ## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - (This PR): apache#18451 ## Changes in this PR This PR refactors state management in HashJoinExec to make filter pushdown more efficient and prepare for pushing down membership tests. - Refactor internal data structures to clean up state management and make usage more idiomatic (use `Option` instead of comparing integers, etc.) - Uses CASE expressions to evaluate pushed-down filters selectively by partition Example: `CASE hash_repartition % N WHEN partition_id THEN condition ELSE false END` --------- Co-authored-by: Lía Adriana <[email protected]> (cherry picked from commit 5b0aa37)

… on the size of the build side (apache#18393) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393 (*this PR*). There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 As those are merged I will rebase this PR to keep track of the "remaining work", and we can use this PR to explore big picture ideas or benchmarks of the final state. (cherry picked from commit c0e8bb5)

* Refactor InListExpr to support structs by re-using existing hashing infrastructure (apache#18449) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - (This PR): apache#18449 (depends on apache#18448) - apache#18451 - Enhance InListExpr to efficiently store homogeneous lists as arrays and avoid a conversion to Vec<PhysicalExpr> by adding an internal InListStorage enum with Array and Exprs variants - Re-use existing hashing and comparison utilities to support Struct arrays and other complex types - Add public function `in_list_from_array(expr, list_array, negated)` for creating InList from arrays Although the diff looks large most of it is actually tests and docs. I think the actual code change is a negative LOC change, or at least negative complexity (eliminates a trait, a macro, matching on data types). --------- Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> (cherry picked from commit 486c5d8) * feat: Add evaluate_to_arrays function (apache#18446) ## Which issue does this PR close?  - Closes apache#18330 . ## Rationale for this change  Reduce code duplication. ## What changes are included in this PR?  A util function replacing many calls which are using the same code. ## Are these changes tested?  No logic should change whatsoever, so each area which now uses this code should have it's own tests and benchmarks unmodified. ## Are there any user-facing changes?   Yes, there is now a new pub function. No other changes to API. --------- Co-authored-by: Martin Grigorov <[email protected]> (cherry picked from commit 76b4156) * Refactor state management in `HashJoinExec` and use CASE expressions for more precise filters (apache#18451) ## Background This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393. There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - (This PR): apache#18451 ## Changes in this PR This PR refactors state management in HashJoinExec to make filter pushdown more efficient and prepare for pushing down membership tests. - Refactor internal data structures to clean up state management and make usage more idiomatic (use `Option` instead of comparing integers, etc.) - Uses CASE expressions to evaluate pushed-down filters selectively by partition Example: `CASE hash_repartition % N WHEN partition_id THEN condition ELSE false END` --------- Co-authored-by: Lía Adriana <[email protected]> (cherry picked from commit 5b0aa37) * Push down InList or hash table references from HashJoinExec depending on the size of the build side (apache#18393) This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in apache#17171. A "target state" is tracked in apache#18393 (*this PR*). There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - apache#18448 - apache#18449 (depends on apache#18448) - apache#18451 As those are merged I will rebase this PR to keep track of the "remaining work", and we can use this PR to explore big picture ideas or benchmarks of the final state. (cherry picked from commit c0e8bb5) * fmt * replace HashTableLookupExpr with lit(true) in proto serialization (apache#19300) *errors* when serializing now, and would break any users using joins + protobuf. --------- Co-authored-by: Adrian Garcia Badaracco <[email protected]> Co-authored-by: David Hewitt <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Emily Matheys <[email protected]> Co-authored-by: Martin Grigorov <[email protected]>

github-actions bot added physical-expr Changes to the physical-expr crates common Related to common crate physical-plan Changes to the physical-plan crate labels Nov 2, 2025

adriangb mentioned this pull request Nov 2, 2025

Refactor InListExpr to support structs by re-using existing hashing infrastructure #18449

Merged

adriangb force-pushed the refactor-create-hashes branch from 4737894 to 57f98ad Compare November 2, 2025 20:45

adriangb mentioned this pull request Nov 2, 2025

Push down InList or hash table references from HashJoinExec depending on the size of the build side #18393

Merged

adriangb mentioned this pull request Nov 3, 2025

Push down entire hash table from HashJoinExec into scans #17171

Closed

adriangb mentioned this pull request Nov 3, 2025

Refactor state management in HashJoinExec and use CASE expressions for more precise filters #18451

Merged

adriangb requested a review from 2010YOUY01 November 3, 2025 18:36

alamb mentioned this pull request Nov 4, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-03 #18486

Closed

53 tasks

martin-g reviewed Nov 6, 2025

View reviewed changes

datafusion/common/src/hash_utils.rs Show resolved Hide resolved

datafusion/common/src/scalar/mod.rs Outdated Show resolved Hide resolved

martin-g approved these changes Nov 6, 2025

View reviewed changes

alamb mentioned this pull request Nov 7, 2025

Use generics in create_hashes pydantic/datafusion#42

Merged

alamb reviewed Nov 7, 2025

View reviewed changes

alamb approved these changes Nov 7, 2025

View reviewed changes

adriangb force-pushed the refactor-create-hashes branch from e50907a to a14159a Compare November 7, 2025 16:23

Refactor create_hashes to accept array references

da34dac

Co-authored-by: Andrew Lamb <[email protected]>

adriangb force-pushed the refactor-create-hashes branch from a14159a to da34dac Compare November 7, 2025 16:36

adriangb added 2 commits November 7, 2025 10:39

add docstring

172f313

fix merge

976d2cd

adriangb added this pull request to the merge queue Nov 7, 2025

Merged via the queue into apache:main with commit a899ca0 Nov 7, 2025
32 checks passed

adriangb deleted the refactor-create-hashes branch November 7, 2025 18:06

-pub trait AsDynArray {
+/// Something that can be returned as a `&dyn Array`
+///
+/// For some reason we can't use `AsRef<dyn Array>` because
+/// it is not implemented for`&dyn Array`
+pub trait AsDynArray {

Refactor create_hashes to accept array references #18448

Refactor create_hashes to accept array references #18448

Uh oh!

Conversation

adriangb commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Changes in this PR

Uh oh!

2010YOUY01 commented Nov 3, 2025

Uh oh!

adriangb commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

2010YOUY01 commented Nov 3, 2025

Uh oh!

adriangb commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 7, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 7, 2025

Uh oh!

adriangb commented Nov 7, 2025

Uh oh!

alamb commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alamb commented Nov 7, 2025

Uh oh!

alamb commented Nov 7, 2025

Uh oh!

alamb commented Nov 7, 2025

Uh oh!

adriangb commented Nov 7, 2025

Uh oh!

alamb commented Nov 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adriangb commented Nov 2, 2025 •

edited

Loading

adriangb commented Nov 3, 2025 •

edited

Loading

adriangb commented Nov 3, 2025 •

edited

Loading

alamb commented Nov 7, 2025 •

edited

Loading