A token is a sub-word chunk (roughly 3-4 English characters). Tokenizers like BPE (GPT) or SentencePiece (Llama, Gemma) split text into integer IDs the model actually consumes.
How it works
The tokenization process begins with a learned merge table that identifies and combines the most frequent byte pairs in a given text corpus. This means that common words, like 'the' or 'and', can be represented as a single token, while less frequent words, technical jargon, or non-English text might be broken down into multiple tokens. For example, the word 'tokenization' could be split into 'token' and 'ization' if those segments appear frequently enough in the data. This method allows the model to efficiently manage its vocabulary size while still being able to represent a wide range of inputs.
Once the text is tokenized, each token is assigned a unique integer ID that the model can process. This is crucial because the model does not understand text directly; it operates on these integer representations. Everything from pricing to context limits and latency is measured in tokens rather than words, which can lead to significant differences in how we think about input size and processing time. For instance, if a model has a context limit of 2048 tokens, that translates to roughly 1500 words in English, but the actual count can vary greatly depending on the content being processed.
In practice, tokenization impacts how models like GPT or Llama interpret and generate text. For example, if you input a sentence that includes a mix of English and code, the tokenizer will break it down into tokens according to its learned patterns. This means that the model can handle a wide variety of inputs, including emojis and programming syntax, without running into issues with unknown words or characters. Without this tokenization process, the model would struggle to understand or generate text that deviates from its training data.
- Input text is processed by tokenization algorithm.
- Common words become single tokens, rare words split into multiple.
- Token IDs are generated for each token.
- Model consumes token IDs as input.
The software-engineering analogue
Think of tokenization as similar to the process of lexing in programming languages, where the source code is broken down into tokens before parsing. In traditional lexing, you define specific rules to identify tokens based on a hand-written grammar. However, with tokenization in NLP, the vocabulary is not predefined; instead, it is statistically learned from a large corpus of text. This means that the boundaries between tokens can be less obvious and are influenced by the language and context of the data, making it a more flexible but complex process.
Why not the obvious alternative
A common initial approach might be to use character-level models, which treat each character as a separate token. While this method allows for complete coverage of any input, it significantly increases the sequence length—up to four times longer than using sub-word tokens. This added length can slow down processing and increase computational costs, making it inefficient for large-scale applications.
On the other hand, using pure word-level tokenization presents its own challenges. Such models struggle with novel words, misspellings, or domain-specific jargon that were not present in the training data. Sub-word tokenization strikes a balance by providing a fixed vocabulary that can still encode any input, including emojis and code snippets, without the downsides of either extreme.
When to use which
In most cases, you won't have to choose your tokenizer; it comes predefined with the model you are using. However, you need to understand how it affects your input. For example, when budgeting for costs or context limits, remember that you can expect around 1.3 tokens per word in English text, but this number can rise significantly for formats like JSON, code, or languages like Chinese. It's wise to use tools like tiktoken to count tokens before finalizing your input, ensuring you stay within the model's limits.
In production: when it breaks
Why it happens. This can occur due to improper tokenization, where the input text is not split correctly into tokens. For example, if the tokenizer fails to recognize common sub-words or misrepresents rare words, the model may struggle to generate coherent outputs, leading to high perplexity.
- Check the tokenization output for a sample input text and compare it to expected tokens.
- Measure the token frequency distribution to identify any unusual sparsity or missing tokens.
- Log the perplexity scores for different subsets of the validation data to see if specific tokens or phrases are causing spikes.
What it delivers
Tokenization provides a clear framework for understanding the costs associated with processing text, how truncation works, and why simply adding more text isn't always a viable solution. By grasping the nuances of tokenization, you can optimize your model's performance and manage its limitations more effectively.