Full BPE tokenization with trained merge rules. Both vocabularies are trained on the same Sinhala-English corpus, the difference is only in how text is split before BPE runs: grapheme clusters vs Unicode codepoints.
sin_eng, codepoints for sin_eng_gpt2),
then the lowest-ranked merge pair is applied repeatedly, exactly matching
BPETokenizer._bpe() in Python.
sin_eng and sin_eng_gpt2 are both trained from scratch on the
same Sinhala–English corpus. sin_eng_gpt2 is not the original
GPT-2 vocabulary. It uses the GPT-2 pre-tokenization strategy (regex word splitting,
codepoint symbols) but with merge rules learned entirely from our training data.
Same corpus, same vocab size, different atomic unit.
Ġ encodes a leading space so word
boundaries are preserved without a separate token.