context : perform output reorder lazily upon access after sync

[ggerganov]

ref #14795 (comment)

After processing a batch, remember the indices that we have to swap and apply the data swap (of logits and embeddings) upon access via llama_get_...

[slaren]

Can we avoid the reorder entirely if we drop llama_get_logits in favor of llama_get_logits_ith?

[ggerganov]

Hm, I think llama_get_logits_ith also requires the reorder?

[ggerganov]

Ah, got it. Let me see.

[ggerganov]

Yes, we can "redirect" the index on-the-fly using the output_ids. Seems like a good idea.

Do it in a follow-up PR, so we can patch the current issue for now?

[compilade]

Hm, I think llama_get_logits_ith also requires the reorder?

Only because the out_ids are sorted, otherwise it wouldn't.

llama.cpp/src/llama-context.cpp

Lines 1221 to 1225 in e4868d1

	std::fill(output_ids.begin(), output_ids.end(), -1);
	for (uint32_t i = 0; i < n_outputs; ++i) {
	output_ids[out_ids[i]] = i;
	}

If llama_get_logits and llama_get_embeddings are removed, then the sorting and output reordering can be removed altogether.

(Right, this comment is redundant with #14853 (comment) and #14853 (comment))

[compilade]

Actually, because of how llama_get_logits_ith is called in llama-perplexity, sorting is still required.

llama.cpp/tools/perplexity/perplexity.cpp

Line 612 in e4868d1

const float * all_logits = num_batches > 1 ? logits.data() : llama_get_logits_ith(ctx, seq*n_ctx + first);

The above assumes the logits after the ith one are in the correct order.

There might need to be an API for logits ranges.

[ggerganov]

This usage of llama_get_logits_ith() is technically wrong because it assumes the logits are sequential, but this is not stated in the API. So I think it is OK to break it.

[slaren]

It is indirectly stated in the comment that says Equivalent to: llama_get_logits(ctx) + ctx->output_ids[i]*n_vocab. The examples would need to be updated, but that should be an easy problem to solve, the bigger problem is that it may be break silently 3rd party code that we have no control over.