Skip to content

[SPARK-53124][SQL] Prune unnecessary fields from JsonTuple #51843

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

wangyum
Copy link
Member

@wangyum wangyum commented Aug 5, 2025

What changes were proposed in this pull request?

This PR enhances the GenerateOptimization rule in Spark SQL Catalyst to improve the pruning of unnecessary fields from JsonTuple generators.

Explicitly handled these cases:

  • No generator outputs used: remove the Generate node.
  • Some generator outputs used: prune the JsonTuple to keep only necessary fields.

Example:

SELECT f2
FROM (SELECT '{"f1": "Spark", "f2": 2025, "f3": 8}' AS json) test
LATERAL VIEW json_tuple(json, 'f1', 'f2', 'f2') jt AS f1, f2, f3

Before this PR:

== Optimized Logical Plan ==
Project [f2#2]
+- Generate json_tuple({"f1": "Spark", "f2": 2025, "f3": 8}, f1, f2, f2), false, jt, [f1#1, f2#2, f3#3]
   +- OneRowRelation

After this PR:

== Optimized Logical Plan ==
Generate json_tuple({"f1": "Spark", "f2": 2025, "f3": 8}, f2), false, jt, [f2#2]
+- OneRowRelation

Why are the changes needed?

Prune unnecessary JSON fields, reducing data processing overhead and improving query performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added unit tests covering all scenarios: no outputs used, some outputs used, and all outputs used.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Aug 5, 2025
@wangyum
Copy link
Member Author

wangyum commented Aug 5, 2025

cc @cloud-fan

}.unzip
p.withNewChildren(Seq(g.copy(
generator = JsonTuple(originJsonTuple.children.head +: newJsonExpressions),
unrequiredChildIndex = Nil,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we didn't do this for the ExplodeBase case, why is it needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reading SPARK-21657, we can remove unrequiredChildIndex = Nil here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants