Token
The smallest piece of text into which an AI model splits language; a word, word fragment or punctuation mark that the model processes, and on which costs and limits are calculated.
By Tanguy De Keyzer · Founder & digital strategist
A token is the smallest piece of text into which an AI model splits language in order to process it. A token is not always a whole word: it can be a word, a word fragment or a punctuation mark. A large language model reads and generates text token by token, and everything you enter or receive is first split into tokens.
How does tokenisation work?
Before the model processes your prompt, it chops the text into tokens using a fixed scheme. Short, common words are often a single token, while longer or rarer words are cut into several pieces. As a rule of thumb, a token corresponds roughly to a short word fragment, and an ordinary paragraph quickly contains several hundred tokens. The model then predicts the next token each time, and this is how the answer builds up step by step.
Why tokens matter
Tokens are the unit of measurement behind both costs and limits. AI providers usually charge per token, for both your input and the generated output. In addition, every model has a context window: the maximum number of tokens it can take in at once. Anyone who supplies long documents or a lot of background hits that limit faster and pays more. With generative AI at scale, this adds up.
What it means for B2B companies
If you deploy AI in processes or automations, token consumption partly determines your cost per action. Short, focused prompts and leaving out superfluous context keep costs manageable without sacrificing quality. Steer on what counts: a process that delivers customers and revenue, not on squeezing every prompt as cheaply as possible. The technology is a lever, and tokens are simply the fuel gauge.
From theory to growth.
We turn Token into measurable results for your business.