olemus
analüüsitava tekstivoo tükeldamine sõnedeks

Wiktionary, tokenize, verb:
1. (transitive, computing) to reduce to a set of tokens by lexical analysis

ülevaateid
https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization
https://hackthology.com/how-to-tokenize-complex-strings-with-lexmachine.html

vt ka
- pöördsõnestus