[Bug]: vector index benchmark performance is lower than pgvector

[heni02]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

b85d757

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

mo vs pgvector vector index benchmark test result:

and also see file:https://doc.weixin.qq.com/sheet/e3_AYYAgwazACsB4L72hnoQ1eAEU31N6?scode=AJsA6gc3AA8QPAY9JuAYYAgwazACs

from the test data, the following performance issues have been summarized：

1. Comparison of query performance with and without index: mo almost has no difference QPS in sift 128dim 1million query with and without index, but there is a 6 times difference in PGvector between the two; The indexed performance of mo 960dim is significantly better than that without indexes

2. QPS performance comparison: sift 128dim and gist960 1million， Mo recall is basically same as pgvector, but QPS is 30-40 times lower than pgvector

3. Performance impact of different lists: Mo index recall which lists=500 is lower than lists=1000, but there is no significant improvement in QPS. However, pgvector QPS which lists=1000 is significantly better than lists=500

4. Index creation performance: Sift 128 1million mo takes 10 times longer to create an index than pgvector; Gist 960 1million mo takes 25 times longer to create an index than pgvector

Expected Behavior

No response

Steps to Reproduce

1.use benchmark tool: git clone -b sift128 https://github.com/arjunsk/mo-benchmark-test.git
2.run benchmark tool with sift128dim and gist960dim dataset

Additional information

No response

[arjunsk]

Currently tracked in #14610 . Will update this issue once the provided issue is resolved.

[arjunsk]

No progress.

[arjunsk]

1. For KNN QPS: //OK
   Waiting for this issue to be resolved: [Bug]: Problem with RuntimeFilter of SEMI join, when we use 2 conditions in OnList #15196
   Once that is fixed, we should get around 50-60 QPS for SIFT128

2. Create Index duration // Needs work
   Made some modifications to bring down the duration to 5-6 mins for SIFT128. Doing some more analysis.

3. Insert before the Index is Created // OK
   Inserts SIFT128 data in 330 secs.

4. Insert after Index is Created // Needs work
   This benchmark is required. Based on my local benchmark we take 30 minutes to reinsert the same SIFT128 data after the index is created for the table. Need more analysis.

[arjunsk]

KNN can be improved after solving this: #15572

Create Index is improved after merging this: #15573

[arjunsk]

Note: Please take the latest pull of https://github.com/arjunsk/mo-benchmark-test/tree/master_index

1. SIFT 128 dataset

• With PK

Load (1million) without INDEX	30sec
Create Index List = 500	3min 41 sec
KNN QPS (k = 5) with INDEX	52
KNN recall (k=5) with INDEX	0.7013
Reinsert (2million) with INDEX	2799 secs
Insert (1million) without INDEX	1292 secs

• Without PK

Load (1million) without INDEX	30sec
Create Index List = 500	3mins 59sec
KNN QPS (k = 5) with INDEX	32
KNN recall (k=5) with INDEX	0.7162
Reinsert (2million) with INDEX	2504secs
Insert (1million) without INDEX	338 secs

[arjunsk]

Hi @heni02 , most of the optimizations have been added to the master. Kindly verify the performance change.