-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Fixed: NameError during evalutation of llamaindex query engine #2331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
1e1f601
13f87ec
c93cf60
553b397
bbd45b1
77f34f5
d5319f4
f1d9061
98fd4c4
3e3a64e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,10 @@ | ||
from __future__ import annotations | ||
|
||
import logging | ||
import math | ||
import typing as t | ||
|
||
from ragas.dataset_schema import EvaluationDataset, SingleTurnSample | ||
from ragas.dataset_schema import EvaluationDataset, EvaluationResult, SingleTurnSample | ||
from ragas.embeddings import LlamaIndexEmbeddingsWrapper | ||
from ragas.evaluation import evaluate as ragas_evaluate | ||
from ragas.executor import Executor | ||
|
@@ -18,10 +19,10 @@ | |
BaseEmbedding as LlamaIndexEmbeddings, | ||
) | ||
from llama_index.core.base.llms.base import BaseLLM as LlamaindexLLM | ||
from llama_index.core.base.response.schema import Response as LlamaIndexResponse | ||
from llama_index.core.workflow import Event | ||
|
||
from ragas.cost import TokenUsageParser | ||
from ragas.evaluation import EvaluationResult | ||
|
||
|
||
logger = logging.getLogger(__name__) | ||
|
@@ -82,8 +83,15 @@ def evaluate( | |
retrieved_contexts: t.List[t.List[str]] = [] | ||
results = exec.results() | ||
for r in results: | ||
responses.append(r.response) | ||
retrieved_contexts.append([n.node.text for n in r.source_nodes]) | ||
# Handle failed jobs which are recorded as NaN in the executor | ||
if isinstance(r, float) and math.isnan(r): | ||
responses.append("") | ||
retrieved_contexts.append([]) | ||
else: | ||
# Cast to LlamaIndex Response type for proper type checking | ||
response = t.cast("LlamaIndexResponse", r) | ||
|
||
responses.append(response.response or "") | ||
|
||
retrieved_contexts.append([n.get_text() for n in response.source_nodes]) | ||
|
||
# append the extra information to the dataset | ||
for i, sample in enumerate(samples): | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's better to fail loudly than silently.
If we still need to pass through, better to keep
None
. The later metrics can skipNone
or handle them explicitly.