Skip to content

Conversation

dranikpg
Copy link
Contributor

@dranikpg dranikpg commented May 18, 2023

Context:

This is a preparation step before adding basic indices to core/search. Indices will be managed per shard, need to be constructed when the index is created with ft.create and need to be updated when new entries are added/deleted. Indices will store integer codes (compact & faster) instead of keys for documents.

Goal:

  • Spread out search_family parts into multiple files to make room for bigger future changes
  • Support accessible per-shard state that will at first store only document key <-> integer id mapping. No updates

@dranikpg dranikpg changed the title [WIP] Search document ids [WIP] Search: Assign document ids May 18, 2023
dranikpg added 3 commits May 25, 2023 16:07
Signed-off-by: Vladislav Oleshko <[email protected]>
Signed-off-by: Vladislav Oleshko <[email protected]>
@dranikpg dranikpg changed the title [WIP] Search: Assign document ids [WIP] Search: Basic shard state May 25, 2023
Signed-off-by: Vladislav Oleshko <[email protected]>
@dranikpg dranikpg marked this pull request as ready for review May 27, 2023 13:53
@dranikpg dranikpg requested a review from romange May 27, 2023 13:53
@dranikpg dranikpg changed the title [WIP] Search: Basic shard state Search: Basic shard state May 27, 2023
struct BaseAccessor : public search::DocumentAccessor {
using FieldConsumer = search::DocumentAccessor::FieldConsumer;

virtual SearchDocData Serialize() const = 0;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remind what Serialize is used for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Convert underlying type to a map<string, string> to be sent as a reply, added a comment

@dranikpg
Copy link
Contributor Author

Fixed comments.

I also did some naming changes, because there will be around 5 depth levels and each one can be called index. Randomly adding words like shard or search doesn't make it more clear 🙂

I suggest calling "logical indices" just document indices (those defined by ft.create SCHEMA PREFIX...). This indicates that they don't perform search, but just keep track of documents.

Shard prefix just means its per shard.

Now its like this:

DocIndex , ShardDocIndices -> ShardDocIndex -> [in pr2] -> FieldIndices -> BaseIndex -> e.g. NumericIndex

Signed-off-by: Vladislav Oleshko <[email protected]>
: query{make_unique<AstExpr>(ParseQuery(query))} {
bool SearchAlgorithm::Init(string_view query) {
try {
query_ = make_unique<AstExpr>(ParseQuery(query));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make a comment explaining why do you try/catch.
catch the exception thrown by parser, ideally vlog it as well.

romange
romange previously approved these changes May 27, 2023
Signed-off-by: Vladislav Oleshko <[email protected]>
@dranikpg
Copy link
Contributor Author

@romange Please re-approve, I want to rebase on this

@dranikpg dranikpg merged commit cc067de into dragonflydb:main May 28, 2023
@dranikpg dranikpg deleted the search-ids branch June 5, 2023 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants