[Question]: HNSW Performance Changes in Nightly Build: Query Speedup vs Recall Drop

[woshichengpeng]

Describe your problem

Summary

We observed significant performance improvements but also recall degradation when comparing the stable version (v0.6.0-dev3) with the nightly build. The query latency improved by ~81.5% but recall dropped by ~7.7%. We're seeking clarification on whether these changes are expected behavior.

Background

During benchmarking of HNSW performance between Infinity versions, we compared:

• Stable version: v0.6.0-dev3
• Nightly build: Latest nightly version (docker id: 48932341cd18)

Performance Changes Observed

Query Performance Improvements

• Query latency: ~81.5% improvement (9.6ms → 3.0ms average)
• P95 latency: ~68.5% improvement (11.1ms → 3.5ms)
• P99 latency: ~65.1% improvement (12.6ms → 4.4ms)

Recall Accuracy Changes

• Recall drop: ~7.7% degradation (0.912 → 0.842)
• ID consistency: ~80% of returned IDs remain the same
• Similarity scores: More accurate in nightly build (old version had systematic calculation errors)

Import Performance

• Import speed: ~40% slower in nightly build
• Memory usage: Optimized in nightly build

Technical Details

• Dataset: NYTimes 256-dimensional vectors (~290K vectors)
• Dataset download: https://ann-benchmarks.com/nytimes-256-angular.hdf5
• Index Type: HNSW with M=16, ef_construction=600, ef=600
• Metric: Cosine similarity
• Batch size: 8192 (same for both versions)

Test Configuration

{
    "name": "infinity_nytimes",
    "app": "infinity",
    "host": "127.0.0.1:23817",
    "data_path": "datasets/NYTimes/nytimes-256-angular.hdf5",
    "insert_batch_size": 8192,
    "topK": 100,
    "mode": "vector", 
    "schema": {
        "embeddings": {"type": "vector, 256, float"}
    }, 
    "vector_size": 256,
    "vector_name": "embeddings",
    "metric_type": "cosine",
    "index": {
        "embeddings": {
            "type": "HNSW",
            "index_params": {
                "M": 16, 
                "ef_construction": 600,
                "metric": "cosine",
                "encode": "lvq"
            },
            "query_params": {
                "ef": 600
            }
        }
    },
    "query_path": "datasets/NYTimes/nytimes-256-angular.hdf5",
    "batch_size": 8192,
    "ground_truth_path": "datasets/NYTimes/nytimes-256-angular.hdf5"
}

Client Fix Applied

During our testing, we discovered and fixed an issue in the benchmark client (python/benchmark/clients/infinity_client.py):

Issue: The ef parameter from query_params was not being sent per query in the original client code.

Fix: Applied the following patch to properly handle query parameters:

diff --git a/python/benchmark/clients/infinity_client.py b/python/benchmark/clients/infinity_client.py
index 7d9ffea12..af9a72d15 100644
--- a/python/benchmark/clients/infinity_client.py
+++ b/python/benchmark/clients/infinity_client.py
@@ -37,6 +37,7 @@ class InfinityClient(BaseClient):
         self.data_mode = self.data["mode"]
         self.path_prefix = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
         self.table_objs = list()
+        self.query_options_ = {}
 
     def _parse_index_schema(self, index_schema):
         indexs = []
@@ -44,10 +45,9 @@ class InfinityClient(BaseClient):
             if value["type"] == "text":
                 indexs.append(index.IndexInfo(key, index.IndexType.FullText))
             elif value["type"] == "HNSW":
-                params = {}
-                for param, v in value["params"].items():
-                    params[param] = str(v)
-                indexs.append(index.IndexInfo(key, index.IndexType.Hnsw, params))
+                # Pass only index-time parameters
+                index_params = {str(k): str(v) for k, v in value.get("index_params", {}).items()}
+                indexs.append(index.IndexInfo(key, index.IndexType.Hnsw, index_params))
             elif value["type"] == "BMP":
                 params = {}
                 for param, v in value["params"].items():
@@ -150,6 +150,20 @@ class InfinityClient(BaseClient):
         else:
             raise TypeError("Unsupport file type!")
 
+    def create_index(self):
+        """
+        Create indexes for the table.
+        """
+        db_obj = self.client.get_database("default_db")
+        table_obj = db_obj.get_table(self.table_name)
+        indexs = self._parse_index_schema(self.data["index"])
+        logging.info(f"Creating {len(indexs)} indexes...")
+        for i, idx in enumerate(indexs):
+            logging.info(f"Creating index: index{i} on column {idx.column_name}")
+            table_obj.create_index(f"index{i}", [idx])
+        logging.info("Finished creating indexes.")
+
+
     def setup_clients(self, num_threads=1):
         host, port = self.data["host"].split(":")
         self.clients = list()
@@ -160,12 +174,22 @@ class InfinityClient(BaseClient):
             self.clients.append(client)
             self.table_objs.append(table_obj)
 
+        # One-time processing of query parameters
+        embedding_column = ""
+        for k, v in self.data["schema"].items():
+            if "vector" in v['type']:
+                embedding_column = k
+                break
+        if embedding_column:
+            query_params_config = self.data["index"][embedding_column].get("query_params", {})
+            self.query_options_ = {str(k): str(v) for k, v in query_params_config.items()}

    def do_single_query(self, query_id, client_id) -> list[Any]:
        result = None
        query = self.queries[query_id]
        table_obj = self.table_objs[client_id]
        if self.data_mode == "vector":
-            res, _ = (
+            res, _, _ = (
                table_obj.output(["_row_id"])
                .match_dense(
                    self.data["vector_name"],
@@ -173,12 +197,13 @@ class InfinityClient(BaseClient):
                    "float",
                    self.data["metric_type"],
                    self.data["topK"],
                    self.query_options_,
                )
                .to_result()
            )
            result = res["ROW_ID"]
        elif self.data_mode == "fulltext":
-            res, _ = (
+            res, _, _ = (
                table_obj.output(["_row_id", "_score"])
                .match_text(
                    "",

Key Changes:

1. Added self.query_options_ = {} in __init__ to initialize query options
2. Fixed HNSW index parameter handling to use index_params instead of params
3. Added query parameter processing in setup_clients to extract query_params from config
4. Added self.query_options_ parameter to match_dense calls to pass query parameters per query
5. Fixed return value unpacking from to_result() calls

This ensures that the ef parameter (and other query parameters) are properly passed to each query, which is crucial for accurate HNSW performance testing.

Questions for the Development Team

1. Expected Behavior: Are these performance/recall trade-offs expected in the nightly build? Is the recall drop a known side effect of the performance optimizations?

2. Root Cause: What specific changes in the HNSW implementation are causing the recall degradation? Is it related to:

   • Memory layout changes affecting search paths?
   • Concurrency improvements altering insertion order?
   • Search algorithm optimizations?

3. Search Path Changes: Are the recall differences due to changes in HNSW search algorithms or just different memory layouts and insertion patterns?

4. Performance vs Accuracy Trade-off: Is this an intentional trade-off where we gain query speed at the cost of some recall accuracy?

5. Future Plans: Are there plans to improve recall while maintaining the performance gains?

6. Configuration Impact: Do different HNSW parameters (M, ef_construction, ef) affect this trade-off differently?

7. Client Fix: Should the query parameter processing fix be incorporated into the main benchmark client?

Verification Results

We've verified that:

• All data is properly inserted (no data loss)
• Missing keys exist in the database
• Similarity calculations are more accurate in nightly build
• The recall drop is due to search/index changes, not data integrity issues
• Query parameters (including ef) are now properly sent per query

Current Impact

• Positive: Significant query performance improvements
• Negative: Recall degradation might affect search quality
• Neutral: More accurate similarity scores
• Fixed: Query parameters now properly applied

Environment

• Stable version: v0.6.0-dev3
• Nightly build: Latest nightly version (docker id: 48932341cd18)
• Dataset: NYTimes 256-angular (https://ann-benchmarks.com/nytimes-256-angular.hdf5)
• Index: HNSW with cosine similarity
• Config: Standard HNSW config (LSG builder not enabled)
• Client: Fixed benchmark client with proper query parameter handling

Additional Context

The nightly build shows improved similarity calculation precision, suggesting the optimizations are generally beneficial. However, the recall drop raises questions about whether this is an acceptable trade-off for the performance gains.

We're simply seeking clarification on whether these observed changes are normal and expected.

---

Labels: performance, hnsw, recall-accuracy, nightly-build, client-fix
Priority: Medium
Component: HNSW Index

[woshichengpeng]

Validation script:

import json
import time
import copy
import h5py
import argparse
import numpy as np
from python.benchmark.clients.infinity_client import InfinityClient

def cosine_similarity(a, b):
    """计算余弦相似度"""
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def verify_similarities(query_vector, result_ids, similarities, data_path, dataset_name="train"):
    """验证相似度计算的正确性"""
    print(f"  验证相似度计算...")
    
    # 读取训练数据
    with h5py.File(data_path, "r") as h5f:
        train_data = h5f[dataset_name][:]
    
    verified_similarities = []
    for i, (result_id, reported_sim) in enumerate(zip(result_ids, similarities)):
        if result_id < len(train_data):
            # 计算真实的余弦相似度
            true_similarity = cosine_similarity(query_vector, train_data[result_id])
            verified_similarities.append(true_similarity)
            print(f"    Top{i+1}: ID={result_id}, 报告相似度={reported_sim:.4f}, 计算相似度={true_similarity:.4f}, 差异={abs(reported_sim-true_similarity):.6f}")
        else:
            print(f"    Top{i+1}: ID={result_id} 超出数据范围")
            verified_similarities.append(0.0)
    
    return verified_similarities

def calculate_recall_with_ground_truth(result_ids, query_id, data_path, top_k=100):
    """使用ground truth计算真正的recall"""
    try:
        with h5py.File(data_path, "r") as h5f:
            if "neighbors" in h5f.keys():
                ground_truth = h5f["neighbors"][query_id][1:top_k+1]  # 跳过第一个元素（通常是查询ID）
                # 计算交集
                intersection = set(result_ids).intersection(set(ground_truth))
                recall = len(intersection) / len(ground_truth)
                return recall, list(intersection), list(ground_truth)
            else:
                print(f"  警告: h5文件中没有neighbors数据集")
                return 0.0, [], []
    except Exception as e:
        print(f"  计算recall时出错: {e}")
        return 0.0, [], []

def parse_args():
    parser = argparse.ArgumentParser(description="对比两个 infinity 版本的性能")
    parser.add_argument("--skip-import", action="store_true", 
                       help="跳过数据导入步骤（如果数据已存在）")
    return parser.parse_args()

# 解析命令行参数
args = parse_args()

# 1. 配置两个 infinity 服务的 host:port
hosts = {
    "old": "127.0.0.1:23817",  # v0.6.0-dev3
    "new": "127.0.0.1:23917",  # test
}

# 2. 选择 config 路径和 hdf5 路径
config_path = "python/benchmark/configs/infinity_nytimes.json"

# 3. 读取 config
with open(config_path, "r") as f:
    base_conf = json.load(f)

# 4. 结果收集
# 修正 data_path 为正确的相对路径
data_path = "python/benchmark/datasets/NYTimes/nytimes-256-angular.hdf5"

# 先检查 hdf5 文件结构
print("检查 hdf5 文件结构...")
with h5py.File(data_path, "r") as h5f:
    print("Available datasets:", list(h5f.keys()))
    # 根据实际数据集名称选择查询数据
    if "test" in h5f.keys():
        query_dataset = "test"
    elif "query" in h5f.keys():
        query_dataset = "query"
    else:
        # 如果没有 test 或 query，使用第一个可用的数据集
        query_dataset = list(h5f.keys())[0]
    print(f"使用查询数据集: {query_dataset}")

results = {}

for tag, host in hosts.items():
    print(f"\n==== 测试 {tag} 版本 ({host}) ====")
    conf = copy.deepcopy(base_conf)
    conf["host"] = host
    # 保存临时 config
    temp_conf_path = f"temp_config_{tag}.json"
    with open(temp_conf_path, "w") as f:
        json.dump(conf, f, indent=2)
    # 5. 初始化 client
    client = InfinityClient(temp_conf_path)
    
    # 6. 清理旧数据（如果未跳过）
    if not args.skip_import:
        print("清理旧数据...")
        try:
            db_obj = client.client.get_database("default_db")
            db_obj.drop_table(client.table_name)
            print(f"已删除旧表: {client.table_name}")
        except Exception as e:
            print(f"清理旧数据时出错（可能表不存在）: {e}")
    else:
        print("跳过数据清理（保持现有数据）...")
    
    # 7. 测量插入速度（如果未跳过）
    if not args.skip_import:
        print("开始导入数据...")
        import_start_time = time.time()
        client.upload()
        import_end_time = time.time()
        import_time = import_end_time - import_start_time
        print(f"导入完成，耗时: {import_time:.2f}秒")
    else:
        print("跳过数据导入...")
        import_time = 0.0
        # 检查表是否存在
        try:
            db_obj = client.client.get_database("default_db")
            db_obj.get_table(client.table_name)
            print(f"表 {client.table_name} 已存在，可以继续查询")
        except Exception as e:
            print(f"错误: 表 {client.table_name} 不存在，请先运行不带 --skip-import 的版本导入数据")
            print(f"错误详情: {e}")
            exit(1)
    
    # 8. 查询
    print("开始查询...")
    client.setup_clients(num_threads=1)
    # 取前20条 query
    with h5py.File(data_path, "r") as h5f:
        queries = h5f[query_dataset][:20]
    sim_results = []
    query_times = []
    
    for i, q in enumerate(queries):
        t0 = time.time()
        table_obj = client.table_objs[0]
        result, extra_result = (
            table_obj.output(["_row_id", "_similarity"])
            .match_dense(
                conf["vector_name"],
                q,
                "float",
                conf["metric_type"],
                conf["topK"],
                client.query_options_,
            )
            .option({"total_hits_count": False})
            .to_pl()
        )
        t1 = time.time()
        query_time = t1 - t0
        query_times.append(query_time)
        
        # 根据实际列名访问数据
        if "ROW_ID" in result.columns and "SIMILARITY" in result.columns:
            # 获取所有返回的ID（用于计算recall）
            all_ids = []
            for j in range(result.shape[0]):
                all_ids.append(result.select("ROW_ID").item(j, 0))
            
            # 获取前5个结果
            top_ids = []
            top_similarities = []
            for j in range(min(5, result.shape[0])):
                top_ids.append(result.select("ROW_ID").item(j, 0))
                top_similarities.append(result.select("SIMILARITY").item(j, 0))
            
            returned_count = result.shape[0]
            print(f"Q{i+1:2d}: latency={query_time:.4f}s, returned_count={returned_count}")
            print(f"  Top5 IDs: {top_ids}")
            print(f"  Top5 Similarities: {[f'{s:.4f}' for s in top_similarities]}")
            
            # 计算ground truth recall
            recall, intersection, ground_truth = calculate_recall_with_ground_truth(all_ids, i, data_path)
            print(f"  Ground Truth Recall: {recall:.4f} ({len(intersection)}/{len(ground_truth)})")
            if len(intersection) > 0:
                print(f"  交集ID: {intersection[:10]}{'...' if len(intersection) > 10 else ''}")
            
            # 验证相似度计算（只对前3个查询进行验证，避免输出过多）
            if i < 3:
                verified_sims = verify_similarities(q, top_ids, top_similarities, data_path)
                print(f"  验证结果: {[f'{s:.4f}' for s in verified_sims]}")
        else:
            print(f"Q{i+1:2d}: latency={query_time:.4f}s, Warning: Expected columns not found. Available: {result.columns}")
            top_ids = [0] * 5
            top_similarities = [0.0] * 5
            all_ids = []
            returned_count = result.shape[0]
            print(f"  returned_count={returned_count}")
            recall = 0.0
            intersection = []
            ground_truth = []
        
        sim_results.append({
            "latency": query_time,
            "top_ids": top_ids,
            "top_similarities": top_similarities,
            "returned_count": returned_count,
            "all_ids": all_ids,
            "recall": recall,
            "intersection": intersection,
            "ground_truth": ground_truth
        })
    
    # 计算统计信息
    avg_query_time = sum(query_times) / len(query_times)
    min_query_time = min(query_times)
    max_query_time = max(query_times)
    
    # 计算平均recall
    recalls = [r["recall"] for r in sim_results]
    avg_recall = sum(recalls) / len(recalls) if recalls else 0.0
    
    results[tag] = {
        "import_time": import_time,
        "avg_query_time": avg_query_time,
        "min_query_time": min_query_time,
        "max_query_time": max_query_time,
        "avg_recall": avg_recall,
        "query_results": sim_results
    }
    
    print(f"\n{tag} 版本统计:")
    if not args.skip_import:
        print(f"  导入时间: {import_time:.2f}秒")
    print(f"  平均查询时间: {avg_query_time:.4f}秒")
    print(f"  最快查询: {min_query_time:.4f}秒")
    print(f"  最慢查询: {max_query_time:.4f}秒")
    print(f"  平均Recall: {avg_recall:.4f}")

# 9. 详细对比输出
print("\n" + "="*80)
print("详细版本对比结果")
print("="*80)

# 性能对比
print(f"\n性能对比:")
if not args.skip_import:
    old_import = results["old"]["import_time"]
    new_import = results["new"]["import_time"]
    import_improvement = ((old_import - new_import) / old_import) * 100
    print(f"  导入速度提升: {import_improvement:.1f}% ({old_import:.2f}s → {new_import:.2f}s)")

old_avg_query = results["old"]["avg_query_time"]
new_avg_query = results["new"]["avg_query_time"]
query_improvement = ((old_avg_query - new_avg_query) / old_avg_query) * 100

print(f"  查询速度提升: {query_improvement:.1f}% ({old_avg_query:.4f}s → {new_avg_query:.4f}s)")

# Recall对比
old_avg_recall = results["old"]["avg_recall"]
new_avg_recall = results["new"]["avg_recall"]
recall_change = ((new_avg_recall - old_avg_recall) / old_avg_recall) * 100 if old_avg_recall > 0 else 0
print(f"  Recall变化: {recall_change:+.1f}% ({old_avg_recall:.4f} → {new_avg_recall:.4f})")

# 详细查询结果对比
print(f"\n详细查询结果对比:")
for i in range(20):
    old_result = results["old"]["query_results"][i]
    new_result = results["new"]["query_results"][i]
    
    latency_improvement = ((old_result["latency"] - new_result["latency"]) / old_result["latency"]) * 100
    recall_diff = new_result["recall"] - old_result["recall"]
    
    print(f"Q{i+1:2d}:")
    print(f"  延迟: {old_result['latency']:.4f}s → {new_result['latency']:.4f}s ({latency_improvement:+.1f}%)")
    print(f"  Recall: {old_result['recall']:.4f} → {new_result['recall']:.4f} ({recall_diff:+.4f})")
    
    # 对比前5个结果
    old_ids = old_result['top_ids']
    new_ids = new_result['top_ids']
    old_sims = old_result['top_similarities']
    new_sims = new_result['top_similarities']
    
    print(f"  Top5 IDs对比:")
    for j in range(5):
        id_match = "✓" if old_ids[j] == new_ids[j] else "✗"
        print(f"    Top{j+1}: {old_ids[j]} → {new_ids[j]} {id_match}")
    
    print(f"  Top5 Similarities对比:")
    for j in range(5):
        sim_change = ((new_sims[j] - old_sims[j]) / old_sims[j]) * 100 if old_sims[j] != 0 else 0
        print(f"    Top{j+1}: {old_sims[j]:.4f} → {new_sims[j]:.4f} ({sim_change:+.1f}%)")

# 统计相似度变化
old_similarities = [r["top_similarities"][0] for r in results["old"]["query_results"]]  # 只统计top1
new_similarities = [r["top_similarities"][0] for r in results["new"]["query_results"]]
avg_old_sim = sum(old_similarities) / len(old_similarities)
avg_new_sim = sum(new_similarities) / len(new_similarities)
overall_sim_change = ((avg_new_sim - avg_old_sim) / avg_old_sim) * 100

print(f"\n总体统计:")
print(f"  平均相似度变化: {overall_sim_change:+.1f}% ({avg_old_sim:.4f} → {avg_new_sim:.4f})")
if not args.skip_import:
    print(f"  导入速度提升: {import_improvement:.1f}%")
print(f"  查询速度提升: {query_improvement:.1f}%")
print(f"  Recall变化: {recall_change:+.1f}%")

# 计算ID一致性统计
total_id_matches = 0
total_id_checks = 0
for i in range(20):
    old_ids = results["old"]["query_results"][i]["top_ids"]
    new_ids = results["new"]["query_results"][i]["top_ids"]
    for j in range(5):
        total_id_checks += 1
        if old_ids[j] == new_ids[j]:
            total_id_matches += 1

id_consistency = (total_id_matches / total_id_checks) * 100
print(f"  ID一致性: {id_consistency:.1f}% ({total_id_matches}/{total_id_checks})") 

[woshichengpeng]

It seems related to the configuration. When I specify the config file, the recall remains the same. However, I'm not sure which config affects HNSW recall.

[JinHai-CN]

1. Please submit the PR to update the benchmark tools
2. You are using LVQ(Locally-adaptive vector quantization.) quantization. Use 'plain' type will get better recall rate.