tokenization

Tokenization Is a Design Choice, Not a Law

A demo note that starts with a basic preprocessing assumption and exposes the system boundary hidden inside it.

• nlp
• tokenization
• mental-models