MEDIUM SEVERITYCONFIRMED
train() crashes with ValueError when vocab_size exceeds mergeable pairs
Package
minbpe
Version
latest
Verified
2026-03-28
Description
minbpe's `BasicTokenizer.train()` crashes when `vocab_size` exceeds the number of unique mergeable byte pairs in the training text. `max()` is called on an empty iterable, raising `ValueError: max() iterable argument is empty`.
Reproduction
from minbpe import BasicTokenizer
BasicTokenizer().train('aaaaaaaaaa', 261)
# ValueError: max() iterable argument is empty