Fixed: NameError during evalutation of llamaindex query engine

[Prigoistic]

Issue Link / Problem Description

• Fixes #2330
• Evaluating a LlamaIndex query engine raised a runtime NameError: EvaluationResult not defined, because it was imported only under t.TYPE_CHECKING. Intermittent LlamaIndex execution failures also led to IndexError during result collection due to mismatched lengths.

Changes Made

• Import EvaluationResult at runtime from ragas.dataset_schema in src/ragas/integrations/llama_index.py.
• Make response/context collection robust:
  • Handle failed executor jobs (NaN placeholders) by inserting empty response/context to maintain alignment with dataset size.
  • Prevent IndexError during dataset augmentation.
• Light defensive checks to ensure stable evaluation even when some query-engine calls fail.

Testing

• Automated tests added/updated

How to Test

• Manual testing steps:

1. Install for local dev: uv run pip install -e . -e ./examples
2. Follow the LlamaIndex integration guide to set up a query_engine and EvaluationDataset: docs
3. Ensure LlamaIndex LLM is configured with n=1 (or unset) to avoid “n values greater than 1 not support” warnings.
4. Run an evaluation that previously failed; it should complete without the NameError and without IndexError during result collection.
5. Optional: run lints uv run ruff check .

References

• Related issues: #2330
• Documentation: LlamaIndex integration how-to (link)

Screenshots/Examples (if applicable)

• N/A

[jjmachan]

hey @Prigoistic I've fixed the CI - could you take a look and see if everything looks good?
we'll merge it in after that 🙂

[anistark]

I think it's better to fail loudly than silently.

If we still need to pass through, better to keep None. The later metrics can skip None or handle them explicitly.

responses.append(None)
retrieved_contexts.append(None)
logger.warning(f"Query engine failed for query {i}: '{queries[i]}'")

[anistark]

This'll be hard on type hints.

Probably better to take from llama_index.core.base.response.schema import Response as LlamaIndexResponse

[anistark]

Make this more explicit?

responses.append(response.response if response.response is not None else "")

[Prigoistic]

@jjmachan yes everything looks good to me

[Prigoistic]

I see no conflicts so far and all the checks has been passed too, you can go further and merge this :)

[anistark]

@Prigoistic I don't see any changes to the comments.

[Prigoistic]

@anistark oh shoot i forgot to push the changes gimme a sec

[Prigoistic]

@anistark pushed the changes as per the comments :) please check it once