HIGH SEVERITYCONFIRMED
9+ metric functions return np.nan as score sentinel — silently poisons aggregations
Package
ragas
Version
0.4.3
Verified
2026-03-28
Description
ragas uses `np.nan` as a sentinel for 'could not compute' across at least 9 distinct metric functions (faithfulness, answer relevance, answer correctness, context precision, context recall, datacompy score, nv_metrics). No caller in `base.py`'s `single_turn_score()` or `single_turn_ascore()` checks for NaN before returning. Any aggregation over a dataset with even one NaN-producing sample silently poisons the entire result — `sum(scores) / len(scores)` returns NaN.
Reproduction
# LLMContextRecall._compute_score([]) -> np.nan
# response = []; denom = len(response) # 0
# score = numerator / denom if denom > 0 else np.nan # -> np.nan
# No exception raised, NaN returned to caller
scores = [0.8, float('nan'), 0.9]
print(sum(scores) / len(scores)) # nan — entire aggregation silently poisoned