How Text Tokenization Works in Neural Networks: From Words to Numbers

Text tokenization is the process that enables a neural network to transform human language into numbers it can understand. Unlike humans, a neural network doesn't "read" sentences in the traditional sense. For the model to process text-composed of words, spaces, and punctuation-it must first be converted into a form suitable for mathematical computation. This is precisely the role of text tokenization, one of the initial steps in handling any input to a language model.

Simply put, the process works as follows: the original text is split into small fragments-tokens-each assigned a unique numerical identifier. These numbers are then converted into vector representations, which are processed by the neural network's layers. Thus, the journey from a regular sentence to data inside the model can be depicted as a chain: text → tokens → IDs → vectors → neural computations.

What is text tokenization and why does a neural network need it?

Text tokenization is the process of breaking an input string into individual elements that the model can process. These elements might be whole words, parts of words, single characters, digits, or punctuation marks.

For example, a human perceives the word "tokenization" as a whole. Depending on the model, the tokenizer may keep it as a single token or split it into several fragments. The same applies to names, technical terms, rare words, and words in various languages.

The reason is straightforward: a neural network cannot directly perform mathematical operations on letters or words-processors work with numbers, so text must first be represented numerically. Tokenization acts as the intermediary layer between human language and the mathematical model.

It's important to note that tokenization is not simply splitting a sentence by spaces. Take the phrase:

The neural network understands text.

A naïve split on spaces would yield three words. In reality, a tokenizer may break it down further-into parts of long words, individual punctuation, or several fragments of a single word. The specific result depends on the algorithm and vocabulary the model uses.

A token should not be viewed as synonymous with a word. Sometimes, a common short word is a single token. But a long or rare word can require two, three, or even more tokens. Conversely, a token might be a punctuation mark, a sequence of digits, or a part of a word.

This means two sentences with the same word count can differ greatly in the number of tokens. Moreover, texts with the same meaning in Russian and English may also have different token counts.

To learn more about what tokens are, why language models need them, and why ChatGPT counts tokens rather than words, see the article Tokens in Neural Networks: What They Are and Why ChatGPT Counts Tokens, Not Words.

From text to tokens

The tokenizer receives the input string and breaks it into a sequence of elements from its vocabulary. Token boundaries don't have to align with word boundaries-a token may be a whole word, a part of a word, or even a single symbol.

From the original string to a sequence of fragments

Imagine the simple phrase:

The neural network analyzes text.

A human immediately sees three words and a period. For a tokenizer, the structure might look different. Hypothetically, it could split the string as follows:

[Neuro] [network] [ analyzes] [text] [.]

This is just an example; actual segmentation depends on the specific tokenizer. One model may keep "neural network" as a single element, while another may split it into two or more parts.

Spaces are also considered by the algorithm. In some systems, a token may include the space before a word. Thus, the same sequence of letters at the start of a string and after another word may be represented as different vocabulary elements.

This approach avoids storing every possible word form in the vocabulary. Instead of millions of individual variants, the model can assemble rare and complex words from more universal fragments.

Why can one word take up several tokens?

The more frequently a certain sequence appeared in the data used to build the tokenizer, the higher the chance it will have its own token. Common words, endings, and character combinations are usually represented more compactly.

For rare words, it's the opposite. For instance, an uncommon technical term, surname, or long compound word may not exist as a whole in the vocabulary. In that case, the tokenizer breaks it down into familiar parts.

Consider the word:

microarchitecture

It could be represented as:

[micro] [architect] [ure]

If there are no large fragments in the vocabulary, the split will be even finer.

The difference between languages is especially noticeable. A tokenizer with good coverage of English sequences may encode English text more compactly than Russian, Japanese, or other languages. Thus, semantically equivalent sentences in different languages can occupy different amounts of space in the model's context window.

Unusual symbols, emojis, digit sequences, and code may also be split in ways that seem unpredictable to a human. For example, a long identifier or random letter combination might use far more tokens than a short, everyday sentence.

What is a tokenizer?

A tokenizer is an algorithm and data set that determines how text is turned into tokens. A crucial part is the vocabulary, where each permissible token is mapped to a unique numerical identifier.

A section of such a vocabulary might look like this:

• "text" → 4812
• "neuro" → 7351
• "network" → 294
• "." → 13

The numbers here are for illustration only; different models have different vocabularies and identifiers.

When a user sends a request, the tokenizer finds suitable vocabulary elements and forms a sequence. If a whole word is not in the vocabulary, the algorithm uses smaller components. As a result, virtually any text can be represented using a restricted set of tokens known to the model.

The size and structure of the vocabulary affect many things: how compactly text is encoded, how different languages are handled, and how many tokens are needed for rare words. Thus, tokenization is not just a technical formality, but a crucial part of a language model's architecture.

BPE and Byte Pair Encoding: how neural networks choose word fragments

One common approach to tokenization is Byte Pair Encoding (BPE). Initially developed as a data compression method, BPE later found use in building language model vocabularies.

The key idea of BPE is to find frequently occurring sequences and merge them into larger elements. This allows the tokenizer to store common words or word parts in full, while assembling rare constructions from several shorter tokens.

If the vocabulary contained only whole words, it would quickly become enormous. For Russian, for example, you'd have to store numerous forms of the same word separately-"нейросеть", "нейросети", "нейросетей", and so on. Plus, new names, terms, product names, and other sequences constantly emerge.

Storing every possible variant individually is inefficient. But splitting all text into single characters is also impractical, as sequences become too long. BPE finds a compromise between these extremes.

The principle can be illustrated with a simplified example: suppose the training data frequently contains the words "neuron", "neural network", "neural", etc. Initially, the algorithm treats them as sequences of individual characters. It then counts which neighboring characters most often occur together. If "n" + "e" appears frequently, it's merged into a new element "ne".

The algorithm repeatedly identifies and merges the most frequent pairs, leading to larger elements like "neuro", which can become a standalone token since it appears in many words. This means words like "neural network", "neuron", and "neurointerface" don't need to be stored in full-the tokenizer can use the shared fragment "neuro" and represent the rest of the word with other tokens.

This approach codes frequent constructions compactly, while rare words absent from the vocabulary are split into smaller known elements. The model isn't limited to a fixed set of familiar words.

It's worth noting that Byte Pair Encoding doesn't mean all modern language models use the exact same tokenization scheme. Variations of BPE and other token construction methods exist. The base units, vocabulary size, merge rules, and how spaces or special symbols are handled may differ.

Thus, the same phrase might be split into a different number of tokens in two models, even if both use similar principles, because their vocabularies were built independently and may highlight different sequences as most useful.

The core task remains the same: representing the nearly infinite variety of human text with a finite set of elements. Once the tokenizer defines these elements, each can be mapped to a number and passed on to the neural network.

How tokens become numbers a neural network can understand

After tokenization, the model receives a sequence of elements, not words. But even tokens themselves cannot be fed directly to a neural network. The next step is to assign each a numerical identifier.

Numerical ID for each token

In the tokenizer's vocabulary, every token has a unique integer. For instance, the phrase:

neural network understands text

might first become:

[neuro] [network] [ understands] [text]

and then a sequence of IDs:

[3817, 9421, 15603, 4812]

The numbers here are illustrative; actual values depend on the tokenizer's vocabulary.

You can think of an ID as the token's row number in a vast table. If "neuro" is assigned 3817, the tokenizer can always use this number to represent it. At this stage, the text is now a numerical sequence, but the neural network still can't extract meaning from the numbers alone.

Why a token's ID isn't enough

The numerical ID doesn't reveal a token's meaning. For instance, if "cat" is 500 and "dog" is 9000, it doesn't imply that "dog" is 18 times larger or more important than "cat".

IDs are for identifying elements in the vocabulary, similar to row numbers in a database-they help find records but say nothing about their content. That's why, before the main processing, each ID is used to retrieve another representation-a set of numbers called a vector or embedding.

How vectors are created

For every token, the model stores a numerical representation with many values. Instead of just an ID like:

3817

the neural network gets something like:

[0.12, -0.47, 0.83, 0.05, ...]

In real language models, these vectors can have hundreds or thousands of components. This isn't just a serial number. During training, model parameters are tuned so that vector representations help the model capture relationships between language elements. Tokens used in similar contexts may receive vectors with certain similarities.

It's not the case that a single coordinate of such a vector literally means "animal", "object", or "positive emotion". The meaning is distributed across many parameters and is understood in the context of the entire model.

Once vectors are obtained, extra information is added to account for the position of elements in the sequence. This is crucial since the meaning of identical words in different orders can change dramatically.

In summary, the full path looks like this:

text → tokenization → tokens → numerical IDs → vectors → neural network processing

For example, a user enters a typical sentence. The tokenizer splits it into parts and replaces each with a number. The model retrieves the corresponding vector representations and analyzes relationships between tokens and their context.

At this stage, human language is finally transformed into mathematical data that the neural network can process.

How tokenization affects how a neural network understands text

Tokenization doesn't determine a text's meaning by itself, but it directly influences how the text is presented to the model. The more conveniently a phrase is split into familiar tokens, the more compactly it fits into the context window, making it easier for the model to process the sequence.

For humans, words like "cat", "kitty", and "kitten" are clearly related. The tokenizer, however, may split them differently-one as a single token, another into two, and a third into several fragments. The neural network must then establish the relationships between these elements based on surrounding context.

This is especially evident with rare terms. A common word, frequently encountered during vocabulary construction, is likely to be represented by a comparatively large token. An unusual technology name, surname, chemical formula, or a recently coined term might be split into many parts.

For instance, a familiar word might look like:

[computer]

Whereas a rare name could become:

[quan] [to] [grav] [imeter] [y]

This doesn't mean the model can't handle rare words-it still receives a sequence of valid tokens and can analyze them in context. However, such representations take up more space and consist of more elements.

The same applies to usernames, website addresses, serial numbers, random character sequences, and code. What seems a short string to a human can become a surprisingly long token sequence.

Text language also matters. The tokenizer's vocabulary is built on a specific dataset, so different languages are represented unevenly. If frequent English fragments are stored as large blocks, but Russian sequences are split more finely, Russian text may require more tokens to convey the same information.

This difference is especially important with long documents. The model's context window is measured in tokens-not pages, characters, or words. Thus, two texts of equal visible length may occupy different amounts of available context.

But tokenization alone doesn't equate to language understanding. When a word is split into several tokens, the model doesn't treat each as a separate human concept. Once converted to vectors, the neural network analyzes the whole sequence and the relationships between its parts.

For example, the word "key" might mean a tool, a water source, a way to decrypt data, or a system access element. The tokenizer doesn't choose the meaning-it simply converts text into a sequence of elements. Determining the correct sense in context is handled by the model at later processing stages.

So, the answer to "how does a neural network understand text" starts with tokenization, but doesn't end there. The tokenizer translates human language into a format suitable for computation, while contextual and relational analysis between tokens happens within the language model itself.

Conclusion

Text tokenization is the first step in transforming ordinary human language into data that a neural network can process. The text is split into tokens, each assigned a numerical ID, then the model receives its vector representation for further computation.

A token is not necessarily a word-it can be a whole word, part of a word, a symbol, or punctuation. Thus, the number of tokens depends not only on text length, but also on language, vocabulary, and the specific tokenizer.

Algorithms like Byte Pair Encoding help balance between an overly large vocabulary of whole words and the inefficiency of splitting text into individual characters. Frequent sequences become larger tokens, while rare words are assembled from smaller fragments.

As a result, the path of text inside a language model can be visualized simply as:

human text → tokenizer → tokens → numerical IDs → vectors → neural network

Understanding this process helps explain why ChatGPT and other language models measure context size in tokens, why texts of equal length may use varying amounts of context, and how an ordinary sentence ultimately becomes a set of numbers ready for mathematical processing.