[Data] - Add support for drop expr() for drop_columns

[goutamvenkat-anyscale]

Description

Right now drop_columns uses map_batches instead of an expression thereby preventing any pushdown optimizations.

Following changes were made:

1. Added DropExpr in Expressions
2. Used [star(), drop(cols)] in drop_columns
3. Updated Projection Pushdown
4. Had to set mongo's client to None because of operator fusion

Before: drop_columns used map_batches → no fusion with Read → different operator structure
After: drop_columns uses expressions → CAN fuse with Read → the datasource needs to be serializable earlier/more frequently

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request is a great improvement. It refactors drop_columns to use an expression-based approach by introducing DropExpr. This change enables drop_columns to participate in logical plan optimizations like projection pushdown, which should improve performance. The implementation is thorough, touching all layers of the expression system from definition to evaluation and optimization. The addition of comprehensive tests, especially the parametrized tests covering various fusion scenarios, gives high confidence in the correctness of this complex change.

[goutamvenkat-anyscale]

pre_renaming

[goutamvenkat-anyscale]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors drop_columns to use an expression-based approach instead of map_batches. This is a significant architectural improvement that enables projection pushdown optimizations for drop_columns. The changes include introducing a new DropExpr, updating the projection fusion and pushdown logic to correctly handle this new expression, and adding comprehensive tests to ensure correctness across various scenarios. The implementation is solid and the new tests are thorough. A potential serialization issue in the Mongo datasource is also fixed. Overall, this is a high-quality contribution that improves the performance and consistency of the Data API.

[aslonnie]

does not seem to require CI review?

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.