MEDIUM SEVERITYCONFIRMED
load() crashes when special token name contains a space
Package
minbpe
Version
latest
Verified
2026-03-28
Description
minbpe's `load()` method parses special tokens by splitting on whitespace. A special token name containing a space produces more than 2 fields when split, raising `ValueError: too many values to unpack (expected 2, got 3)`.
Reproduction
# Writing model file with special token containing a space
# Then calling tokenizer.load('model.model')
# ValueError: too many values to unpack (expected 2, got 3)