Understanding Tokens in Neural Networks: How Language Models Process Text

Tokens in neural networks are small fragments of text that a language model, such as ChatGPT, splits your input into before processing. While a sentence appears as a string of words to a user, for ChatGPT it becomes a sequence of individual tokens converted into numbers.

This is why language model limitations are usually specified in tokens rather than words or characters. The number of tokens determines how much text the model can process at once, how much of a long conversation it can consider, and how lengthy a reply it can generate.

What Are Tokens in Neural Networks?

Token: Explained Simply

A token is a unit of text that a language model works with. You can think of it as a small chunk of a string, to which the system assigns a specific numerical identifier.

A token is not necessarily an entire word. Depending on the tokenizer used, a token may be a short common word, part of a longer word, a string of letters, a number, or a punctuation mark.

For example, the phrase:

"The cat sits on the table."

looks like four words and a period to a human. However, a neural network might split it differently: some words may stay whole, while others are divided into parts. The exact result depends on the model's tokenizer and vocabulary.

After this splitting, each fragment gets a number. Inside its calculations, the neural network no longer works with the string "cat" or "table," but with their numeric representations.

Why a Token Is Not the Same as a Word

It might seem easier to assign each word its own number. But this approach quickly runs into the problem of language's vast variety: the number of possible words is almost limitless.

Take the Russian word "работа" ("work")-it has many forms: "работы," "работами," "работающий," and so on. If each form were a separate dictionary entry, the size would grow rapidly. Names, company titles, new terms, typos, numbers, links, and code complicate things further.

Tokens allow rare and long words to be broken down into familiar parts for the model. This means the neural network can process even sequences that didn't appear as a full word in its vocabulary.

So, "one word equals one token" is incorrect. A short, common word might occupy a single token, while a rare or complex word could use several. The same applies to punctuation and numbers-dates or web addresses may be represented by multiple tokens. For the model, all these are just sequence elements, not distinct human categories like "word," "comma," or "number."

This explains why two texts with the same number of words can differ greatly in token count. The true size of a request is defined by the tokenizer's output-not by the visible word count.

How Text Is Turned Into Tokens

What Happens After You Submit a Prompt

When you send a message to ChatGPT, it's not transmitted in its raw form. First, a tokenizer splits the string into a sequence of tokens:

text → tokenization → tokens → numeric IDs → neural processing

Suppose you write: "How does a neural network work?" To a human, that's three words and a question mark. The tokenizer may split it into several fragments, with token boundaries not always matching word boundaries.

Each token is assigned an identifier from the model's vocabulary-say, token IDs 4217, 853, and 19024. These numbers have no meaning to a human; they're just unique dictionary references. The model then processes this sequence of IDs in further computations.

Why Do Neural Networks Need Tokenization?

A neural network performs mathematical operations and cannot read letters and words as humans do. A phrase like "Today is sunny" is meaningless to a computer until it's converted into numbers.

Assigning a unique number to every possible word doesn't scale: languages evolve with new words, slang, names, technical terms, and more. Also, each word may have many grammatical forms.

Tokenization solves this by using a limited vocabulary of common text fragments. If a word is in the vocabulary, it can be a single token; if not, the tokenizer breaks it into known pieces. This lets the model handle not just normal sentences, but names, numbers, URLs, code, and neologisms.

Why Does Text Get Split This Way?

The token vocabulary is set when the model is created. Frequently seen character sequences are stored as larger elements to shorten token sequences. Rare combinations are split into smaller tokens. So, a common word may be one token, while a complex technical term may be several.

The tokenizer, its vocabulary, and the text's language all affect splitting. The same line might yield a different token count in different models.

For an in-depth look at tokenizers, vocabulary creation, subword splitting, and how text turns into numbers, see How Text Tokenization Works in Neural Networks: From Words to Numbers.

Why Does ChatGPT Count Tokens Instead of Words?

Why Models Use Tokens

Using words as the base units for language models is impractical. Language is too diverse: there are millions of word forms, names, technical terms, abbreviations, and new expressions. Assigning each a unique dictionary item would make the dictionary huge, and unfamiliar words would need special handling.

Tokens offer a more flexible approach. Common words or fragments are stored whole, while rare sequences are built from smaller elements. This lets a single system handle regular text, code, numbers, URLs, and multiple languages.

This method also limits dictionary size. Instead of an infinite list of possible words, the model uses a fixed set of tokens, whose combinations can represent nearly any input.

That's why ChatGPT counts tokens, not words. Word count is convenient for people, but for the model, the important thing is the length of the actual sequence it receives for processing.

How ChatGPT Uses Tokens to Generate Answers

After processing your request, the model receives a sequence of tokens and uses them as context to predict what comes next.

tokens received → context analysis → next token probabilities → select continuation → repeat

For example, with the phrase start "The capital of France is-", the model calculates probabilities for possible next tokens. Some have low probability, while the part matching "Paris" has much higher likelihood.

Once the next token is chosen, it's added to the sequence, and the model repeats the calculation, now considering this new token. This cycle repeats many times. So, the answer is not generated as a whole paragraph at once. The language model builds it token by token, each time choosing the next piece based on the context.

The "next token" isn't always a complete word. It could be part of a word, a punctuation mark, or another text fragment. Only after all tokens are generated does the user see conventional sentences.

What Happens to a Token Inside a Neural Network?

The numeric ID of a token isn't yet suitable for computations. Before further processing, each ID is converted into a set of numbers-a vector representation or embedding. This lets the neural network process tokens in multidimensional space and consider relationships between sequence elements.

The vectors then pass through the model's layers. The network analyzes not just each token in isolation, but also its position and relation to others in the context. For instance, the meaning of a word may depend on nearby elements or those much earlier in the sentence. Thus, the same sequence of characters can be interpreted differently depending on context.

As a result, the model forms an internal context representation and uses it to calculate the likelihood of the next token. The process repeats until the answer is finished or a generation limit is reached.

How Many Characters and Words Are in a Token?

How Many Characters Are in a Token?

A token does not have a fixed length. It can be a single character, several letters, a whole word, or even a longer sequence. So, there's no universal rule like "1 token = 4 characters."

Such estimates are used only for rough calculations. In ordinary English text, a token often covers several characters, but the exact value depends heavily on content and the model's tokenizer.

For instance, a common short word may be a single token, while a rare technical term of the same length might be split into several. Numbers, unusual symbols, and character combinations can also increase token counts. Thus, two sentences of equal character length may not take up the same space in a neural network's context.

Can You Convert Token Count to Words?

The number of words can't be exactly converted to tokens by a single coefficient. Sometimes a word is one token, sometimes two, three, or more.

This is especially obvious with long or rare words. The tokenizer might split a word not found in its vocabulary into known fragments. For example, the technical word "microarchitecture" may look like one word to a human, but inside the model, it's several tokens. The reverse is also possible: a common character combination could be a single token and take up less space than expected by length.

So, values like "1000 tokens ≈ X words" are only rough estimates. For an exact count, you need to run the specific text through the same tokenizer used by the model.

What Determines the Number of Tokens?

The final token count is influenced mainly by the text itself and the tokenizer's vocabulary:

• Language. The same meaning in different languages can require different numbers of tokens, due to writing systems, word structures, and which character sequences are represented most efficiently in the tokenizer's dictionary.
• Word frequency. Common words and phrases are stored as larger tokens, while rare terms, unusual names, or new words may be split into more pieces.
• Numbers. A long number may not be a single token. A sequence of digits can be divided into several tokens, depending on the tokenization system.
• Punctuation and special symbols. Periods, commas, brackets, math signs, and other symbols are also tokenized. Sometimes they are their own tokens, sometimes they're grouped with neighboring fragments.
• URLs and other structured strings. A web address contains letters, dots, slashes, parameters, and special symbols, so even a short link may use many tokens.
• Code. Variable names, operators, indentation, brackets, and other syntax elements are counted. Large code fragments may use context space differently than plain text of similar visual length.
• Formatting. Line breaks, markup, and some service elements are also included in the input sequence and can increase its size.

Why Do Russian and English Texts Use Different Numbers of Tokens?

The difference between languages is not because a neural network "understands one worse" than the other. The reason is in tokenization.

The model's vocabulary is built from huge text corpora. If common English words and word parts occurred more often in the data, they may be encoded as larger tokens. Russian words, in turn, might need to be split into more parts. But this isn't universal-modern tokenizers differ, so the ratio depends on the specific model and text.

This is why you can't judge a large document's size solely by pages, words, or characters. For a neural network, the real text size is defined by the post-tokenization token count.

ChatGPT's Context Window and Token Limit

What Is a Context Window?

Token count measures not only the length of a single message. A language model has a context window-the amount of information it can take into account at once when processing input and generating a reply.

Context may include:

• system instructions
• the user's current request
• previous conversation messages
• inserted text
• contents of uploaded documents
• service information
• already generated parts of an answer

All of these use up tokens. Even a short new request may be processed along with a long message history, already consuming much of the available context.

You can think of the context window as the neural network's workspace. As long as the required information fits inside, the model can consider it when forming the next token.

What Does Token Limit Mean?

Each language model type has a context limit-the maximum number of tokens it can process in a single operation or conversation, based on its architecture and service settings.

This limit cannot be directly translated into a fixed number of pages or words. One document might use tokens efficiently, while another of similar length may use them up much faster due to language, code, numbers, or complex formatting.

It's also important whether the input text and generated reply share the available context or have separate limits, depending on the model and interface.

For instance, if much of the context window is occupied by a long document, conversation history, and instructions, there's less room left for further content. Thus, the maximum theoretical context size does not mean you can always send a request of that length and get an equally long answer.

What Happens in Long Conversations?

As a chat continues, the amount of information grows. If the full conversation no longer fits in active context, the system manages history to keep working within the available limit.

The specific method depends on the service and model: some old messages may drop out of context, or information might be summarized or compressed.

In very long conversations, the neural network may lose track of details discussed much earlier. To the user, the chat looks like an unbroken dialogue, but the model does not always process every message from the very beginning each time it replies.

This is one reason why tokens are closely tied to how well a neural network maintains context in extended conversations.

For more on this process, see Why Neural Networks Forget Long Conversations: How Context, Memory, and Attention Work.

Why Token Limits Matter in Practice

Context limits are especially noticeable when working with large amounts of information.

• Document analysis: The model must fit the file's content, user instructions, and space for its answer into context. The bigger the document, the higher the token usage.
• Code work: Context includes files, comments, variable names, technical instructions, and previous changes. Large projects can't be measured just by line count.
• Long chats: Tokens accumulate with message history. The more information needs to be considered at once, the larger the required context window.
• Large prompts: Detailed instructions can give the model more useful context, but also use up available space. Adding more text doesn't always improve results-it's better to keep only what's truly necessary and remove repetitions.

Tokens are not just a technical counting unit-they define the real amount of information a language model can handle at the same time.

FAQ

1. How many characters are in a token?
   There's no fixed number of characters per token. A token may be a single character, several letters, a word, or another sequence. For rough estimates, people sometimes use an average of a few characters per token, but actual counts vary by text.
   The exact number can only be determined using the tokenizer for the intended model.
2. How many words fit into a ChatGPT token?
   The word-to-token ratio isn't constant. A short, common word may be one token; a long or rare word may be several. So, a 1,000-word text may have different token counts depending on language and vocabulary.
   For large prompts, it's best to focus on token count directly, not try to convert to words by a fixed formula.
3. Are punctuation marks counted as separate tokens?
   Yes, punctuation participates in tokenization, but not every mark becomes its own token. Depending on the tokenizer, a period, comma, bracket, or other symbol may be a separate element or grouped with neighboring characters.
   Spaces, line breaks, mathematical symbols, and other text elements are handled similarly.
4. Why can Russian text use more tokens than English?
   Token count depends on the tokenizer's vocabulary and how efficiently common character sequences are represented.
   If frequent English words and parts appeared more often in training data, they may be encoded as larger tokens. Russian words may need to be split more. However, this isn't a hard rule-modern tokenizers vary by model and text.
5. What happens if the ChatGPT token limit is reached?
   If the required information exceeds the available context window, the model cannot process the whole sequence at once. Depending on the service, some old context may be dropped, data may be processed differently, or the system may limit input or output length.
   This is especially important for long conversations, large files, and code analysis.

Conclusion

Tokens in neural networks are the fundamental elements through which a language model receives and generates text. They don't correspond exactly to words or characters: a word may be a single token or split into several parts.

ChatGPT counts tokens because this format enables it to represent the vast and ever-changing human language with a limited vocabulary. After tokenization, each fragment is assigned a numeric ID, which is then converted into a form suitable for computation inside the neural network.

The number of tokens is practically important: it determines how much text a request takes up, how much conversation history fits in the context window, and how much information the model can consider at once. That's why, when working with long documents, code, or large conversations, it's more useful to look at the tokenized text volume than just the number of pages or words.