Skip to content

Conversation

@haohuaijin
Copy link
Contributor

@haohuaijin haohuaijin commented Dec 10, 2023

Which issue does this PR close?

related to #8374

Rationale for this change

In #8374, we find ProjectionPushdown will incorrectly push projection down and rewrite order by column,
before

select t1.* from t1 join t2 on t1.a = t2.a order by t1.a, t2.b;

after

select t1.* from t1 join t2 on t1.a = t2.a order by t1.a, t1.b;

the sort column change from t1.a, t2.b to t1.a, t1.b.

However, even if we find the root case, we can still not explain why setting the order by column incorrectly results in an empty output. After doing some tests, I discovered that we did not set the preserve_partitioning in SortExec. As a result, SortExec only run one partition(instead of all partition), leading to an empty output.

What changes are included in this PR?

make SortExec's preserve_patitioning consistent before and after projection_pushdown

Are these changes tested?

yes, add .slt test

Are there any user-facing changes?

@github-actions github-actions bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Dec 10, 2023
Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @haohuaijin and @waynexia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants