Standard subword tokenizers either treat orthographic variations of words as unrelated entries or discard them through lossy normalization. The Functionalizer addresses this by factoring these variations into a compositional stream of parametric transformation operators and canonical base tokens.
These operators, encoded in the Unicode Private Use Area, cover casing, 13 types of diacritics, and character repetition. The framework is fully reversible, allowing for lossless reconstruction of the original text.
Evaluation across six natural language and code corpora shows that the Functionalizer enables complete coverage with significantly smaller vocabularies under unconstrained conditions.
Evaluation Results
Sequence length analysis reveals a domain-dependent tradeoff: the method compresses indentation-heavy code sequences but inflates natural-language prose sequences. Preliminary evaluations on 25M parameter GPT-2 scale models indicate drastic improvements in code syntax validity and character perplexity, while maintaining similar text coherence.
Source: https://arxiv.org/abs/2609.15991
