olemus
analüüsitava tekstivoo tükeldamine sõnedeks

Wiktionary;
tokenize, verb
1. (transitive, computing) to reduce to a set of tokens by lexical analysis

ülevaateid
https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization

https://hackthology.com/how-to-tokenize-complex-strings-with-lexmachine.html

vt ka
- pöördsõnestus