[Feature Request] More production ready word matching solution

[Wendong-Fan]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

We used rank_bm25 for BM25 matching, but this library has some limitations and is not efficient for large volumes of data. We need a more production-ready solution, and Elasticsearch could be a viable option to explore.

Solution

No response

Alternatives

No response

Additional context

No response

[JINO-ROHIT]

are you referring to lexical search/keyword based search? is it worth exploring bm25?[https://github.com/xhluca/bm25s]

is there a bechmark we have to eval different techniques?

[Wendong-Fan]

Hey @JINO-ROHIT , sorry for the late reply. Yes, this issue is about enhancing our keyword-based search solution. I'm not sure if there's an existing benchmark for it, but we need to consider both performance and production readiness—specifically how well it can handle large-scale data efficiently and robustly.

[JINO-ROHIT]

@Wendong-Fan how about bm25s? llamaindex also uses this implementation -

1. https://github.com/xhluca/bm25s
2. https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/retrievers/llama-index-retrievers-bm25/llama_index/retrievers/bm25/base.py

[JINO-ROHIT]

gentle reminder @Wendong-Fan