[BUG] mg_index doesn't write output values when used on single GPU with default parameters

[achirkin]

When built for a single GPU, mg_index doesn't write the outputs during search with default parameters.

The culprit is the sharded-tree-merge search implementation that is designed for even number of GPUs.
With rank == 1 (single GPU) the code for copying the data to output is never reached:

cuvs/cpp/src/neighbors/mg/snmg.cuh

Lines 427 to 438 in 2db8516

	if (remaining <= 1) {
	raft::copy(neighbors.data_handle() + output_offset,
	tmp_neighbors.data_handle(),
	part_size,
	raft::resource::get_cuda_stream(dev_res));
	raft::copy(distances.data_handle() + output_offset,
	tmp_distances.data_handle(),
	part_size,
	raft::resource::get_cuda_stream(dev_res));
	resource::sync_stream(dev_res);
	}

...because it's nested under the following loop:

cuvs/cpp/src/neighbors/mg/snmg.cuh

Lines 367 to 370 in 2db8516

	int64_t remaining = index.num_ranks_;
	int64_t radix = 2;
	while (remaining > 1) {

That is, if one builds and searches an SNMG index using ANN_BENCH tool with default parameters and a single GPU, the recall is always zero.

[viclafargue]

Thanks for catching this, @achirkin. It appears that tree merge was chosen as the default behavior, leaving it up to the user to opt for direct merge when deploying with an odd number of GPUs. As a workaround, we could automatically detect when tree merge is used with an odd GPU count and fall back to direct merge, accompanied by an informative message.

[viclafargue]

Upon closer examination, I found that the odd‐GPU scenario is already handled correctly: the lone GPU without a merge partner simply skips the first merge step (see condition in the code). Further testing revealed that the only remaining issue occurs when running a sharded search on a single GPU. I’ve opened a PR to address this specific edge case : #1094.