Language models

Token

A token is the smallest unit into which a language model breaks down text. Usually that is three to four letters or a short word. The more tokens a model can process at once, the longer the texts it can understand.

← Back to the glossary