Ray Data Q4 Roadmap + Wishlist

[richardliaw]

Description

Ray Data community has been rapidly growing every month. We're putting out here the roadmap for the upcoming quarter.

Feel free to leave any requests or questions below. If there's anything missing, please feel free to leave below!

Read/Write integrations

• Improved Iceberg Support
  • Upsert support:
    • [Data] dataset.write_iceberg to support upserts #53438,
    • [Data] - Iceberg support upsert tables + schema update + overwrite tables #58270
  • Overwrite support:
    • [Data] Improve the write_iceberg API to support the overwrite/create_table write modes #51951
  • Leveraging partition / metadata information:
    • [Data] - Iceberg support predicate & projection pushdown #58286
  • Creating, altering, deleting tables
• Bounded Kafka Support:
  • [Data] Add Bounded Kafka Datasource #58653
  • [Data] Add Kafka as datasource #58592
• write_kafka support:
  • [Data] Add Kafka as a new Datasink #58725
• Turbopuffer Support
• Land DataSourceV2
• Hive support:
  • [Data] Add support for reading from Hive with Ray Data #58253

Expressions

• Exposing core expression ops: [Data] - Ray Data Compute Expressions #58674
  • [Data] Add namespaced expressions that expose pyarrow functions #58465
  • [Data] Enable expressions for grouped with_column in Ray Data #58231
  • [Data] - Add support for drop expr() for drop_columns #58387

Query Planning

• Predicate pushdown: [Data] - Add Predicate Pushdown Rule #58150, [Data] - Predicate Pushdown - Push predicate exprs past eligible operators #58555
• Projection pushdown: [data] support shuffle projection pushdown #58151

Improved streaming support

• map_groups to support iterator outputs: [data] support generator udf for map_groups #58039

Preprocessors

• Transform optimizations
• Migrate Preprocessors to Aggregate V2 : [Data] Use Approximate Quantile for RobustScaler Preprocessor #58371
• Simplify API for custom preprocessors
• Save/Load framework for preprocessors: [Data] Add serialization framework for preprocessors #58321

Training Dataloading

• Improve shuffle flexibility (i.e., shuffle metadata at row groups or row level)
• Collation off training thread: [Data] Support Exact Batch Size Enforcement in map_batches to Enable Collate Functions in Ray Data Pipeline #58837
• Proper patterns for mixing dataloaders

Shuffle scalability envelope

We plan to publish a scalability envelope for Ray Data and a roadmap for improving scalability to make Ray Data competitive with other data processing frameworks.

LLM integrations

• Multimodal Data Processors: [data][llm] Introduce generic multimodal preparation #58260
• Stage-based configuration: [data][llm] Ray Data LLM Config Refactor #58298
• Tool-calling: [data][llm] Support tool calling in Ray Data LLM #58091

Preview for 2026

• External Shuffle Service: [Data] Support integration with Apache Celeborn #58687
• Unbounded data source support: [Data] Unbounded Data Sources #40513
• Windowed Expressions
• Integration with Ibis