Merge support for tiered index #1155
Conversation
benfred
added
improvement
github-project-automation
bot
moved this to Todo
in Vector Search, ML, & Data Mining Release Board
|
|
||
| auto merge(raft::resources const& res, | ||
| const std::vector<tiered_index::index<cagra::index<float, uint32_t>>*>& indices) | ||
| -> tiered_index::index<cagra::index<float, uint32_t>> |
There was a problem hiding this comment.
I think ideally we would have merge() return index and, as you have done, use the PIMPL idom to allow the return of either the same index type that was input, or a composite_index.
| * @note: When device memory is sufficient, the dataset attached to the returned index is allocated | ||
| * in device memory by default; otherwise, host memory is used automatically. | ||
| * | ||
| * @note: This API only supports physical merge (`merge_strategy = MERGE_STRATEGY_PHYSICAL`), and |
There was a problem hiding this comment.
Based on our discussion yesterday, we should just remove the "merge strategy" concept all together and support only physical in the "merge" apis. We can still provide a composite that will allow for multiple indexes to be searched concurrently, but don't need to make it part of the merge() apis.
| * @brief merge multiple indices together | ||
| */ | ||
| template <typename UpstreamT> | ||
| auto merge(raft::resources const& res, |
There was a problem hiding this comment.
I wonder- if we are going to rebuild the cagra index anyways to consolidate tiered indices, should we just go ahead and add all the vectors in tier0 to that cagra index too (e.g. should we also do a compaction?). Disregard if this is already what we're doing.
| * @{ | ||
| */ | ||
| /** | ||
| * @brief Merge multiple indices together into a single index |
There was a problem hiding this comment.
Can we flesh out this description a little more to provide more details for the user on exaclty how this does the merge? We should do the same in the c++ description also. Maybe add details about what happens to the vectors in tier 0, what happens to the indexes in tier 1, what are some of the memory considerations (I know there's copies- what are some of the assumptions a user will need to make?)
|
/merge |
rapids-bot
bot
merged commit 699d32c
into
rapidsai:branch-25.10
github-project-automation
bot
moved this from In Progress
to Done
in Elasticsearch + cuVS Team Board
github-project-automation
bot
moved this from Todo
to Done
in Vector Search, ML, & Data Mining Release Board
2 participants
No description provided.