MEDIUM SEVERITYCONFIRMED

AnswerAccuracy.average_scores silently returns NaN when both sub-scores are NaN

Package
ragas
Version
0.4.3
Verified
2026-03-28

Description

ragas's `AnswerAccuracy.average_scores` uses `if score0 >= 0 and score1 >= 0` to detect valid scores. `np.nan >= 0` evaluates to `False` in Python (no exception), so both NaN inputs fall through to `max(nan, nan)` which returns NaN. When both LLM calls exhaust their retry budget, the composite score silently becomes NaN. Even one valid sub-score is discarded: `avg(nan, 0.5)` returns NaN rather than `0.5`.

Reproduction

import numpy as np
def average_scores(score0, score1):
    score = np.nan
    if score0 >= 0 and score1 >= 0:  # np.nan >= 0 is False
        score = (score0 + score1) / 2
    else:
        score = max(score0, score1)  # max(nan, nan) = nan
    return score
print(average_scores(np.nan, np.nan))  # nan
print(average_scores(np.nan, 0.5))    # nan — valid score discarded
← All bugsScan your code →