Skip to content

[pixels-retina, trino, sink] Retina Horizontal Scalability #1182

@AntiO2

Description

@AntiO2

Roadmap:

  1. The Coordinator maintains the list of Retina instance IPs. Each Retina instance can serve multiple tables, and each table can be served by multiple Retina instances. However, the same primary key within a table should not be processed by multiple Retina instances. ([pixels-daemon, common, proto] Implement Retina Heartbeat Tracking and consistent hash #1215 )
  2. Trino should be able to fetch the list of Retina instances for query. ([connector] Enable Multi-Retina Server Support in Connector via Distributed Buffer Splits pixels-trino#161 )
  3. Pixels-Sink should be able to obtain the primary key → Retina instance mapping.([pixels-daemon, common, proto] Implement Retina Heartbeat Tracking and consistent hash #1215 )
  4. Implement a Cache Component for bucketId to RetinaNodeInfo Mapping. ([pixels-cli, common] Implement Bucket-Based Data Classification for Distributed Loading #1218 )
  5. Enhance pixels-cli Load Command for Bucket-Based Classification Mapping. ([pixels-cli, common] Implement Bucket-Based Data Classification for Distributed Loading #1218 )
  6. PixelsRecordReader can choose RetinaService according to file name which include Retina Server name ([pixels-core] Support Multiple Retina Nodes in PixelsRecordReader for Distributed Data Access #1221 )

Sub-issues

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions