MEDIUM SEVERITYCONFIRMED
AnswerAccuracy.average_scores silently returns NaN when both sub-scores are NaN
Package
ragas
Version
0.4.3
Verified
2026-03-28
Description
ragas's `AnswerAccuracy.average_scores` uses `if score0 >= 0 and score1 >= 0` to detect valid scores. `np.nan >= 0` evaluates to `False` in Python (no exception), so both NaN inputs fall through to `max(nan, nan)` which returns NaN. When both LLM calls exhaust their retry budget, the composite score silently becomes NaN. Even one valid sub-score is discarded: `avg(nan, 0.5)` returns NaN rather than `0.5`.
Reproduction
import numpy as np
def average_scores(score0, score1):
score = np.nan
if score0 >= 0 and score1 >= 0: # np.nan >= 0 is False
score = (score0 + score1) / 2
else:
score = max(score0, score1) # max(nan, nan) = nan
return score
print(average_scores(np.nan, np.nan)) # nan
print(average_scores(np.nan, 0.5)) # nan — valid score discarded