Effect of quantization on search metrics

[lannelin]

Suggest that we compare performance of document retrieval with and without quantization. We can do this in isolation to the tiptoe protocol, running this in a similar way to the TFIDF benchmark - searching over a single index.

We should be able to pick up embeddings from existing pre-processing. Each run in the data/ dir should have an embeddings/embeddings subdir with:

• embeddings.npy (dim reduced)
• embeddings_original.npy (original)

unfortuantely, the quantized embeddings are only in the clusters_<n>.txt files (in data/<run>/clusters/)
these can be loaded (and matched to the npy files 🤞- needs to be checked!) with something like (hopefully - writing this on the fly in the issue)

from arc_tiptoe/preprocessing/utils import parse_file

# TODO more efficient to do a pre-allocation
all_inds = []
all_quantized_embs = []

for cluster_fpath in cluster_fpaths:
    indices, quantized_embs, _ = zip(*parse_file(cluster_fpath))
    all_inds.extend(indices)
    all_quantized_embs.extend(quantized_embs)

I believe data/c1970a4c-46c4-8fbe-cf94-d099e24ba206-2000-192.zip can be pulled down locally and used for development. Actual testing should be done over entire corpus.

[lannelin]

ok so it looks like embeddings are scaled and quantized per cluster_*.txt. This means we can't naively stack them back together. It also means that, as embeddings can occur in multiple clusters, we have the same embedding index appear with different embedding values 🙃

I suggest we could either:

1. invert the scaling process by looking at the original embedding values for each cluster, then stack all the embs and evaluate over a single search index
   process here:

   arc-tiptoe/src/arc_tiptoe/search/query_processor.py

   Line 222 in 51e2106

   # Quantize embedding

2. evaluate (still without encryption scheme) as a retrieval over multiple clusters

thoughts @eddableheath @J-Dymond ?

[lannelin]

@J-Dymond I've got a script that outputs csvs but getting an issue analysing them. Could you try running and take a look?
I'm running the new script with

python scripts/quantization_eval_pre.py --data_dir ./data/c1970a4c-46c4-8fbe-cf94-d099e24ba206-2000-192 --query_text_fpath ./test_data/test_queries.tsv --dimred 192

[lannelin]

also @eddableheath is a higher "score" in an output csv better? I'm just outputting the distances there so will need to invert if so.