Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Transformers v5 redesigns how tokenizers work. The big tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can inspect, customize, and train from scratch with far less friction.

TL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. It’s a practical guide for anyone who wants to understand, customize, or train model-specific tokenizers instead of treating them as black boxes.

Table of Contents

• What is Tokenization?
• The Tokenization Pipeline
• Tokenization Algorithms
• Accessing tokenizers through transformers
• The Tokenizer Class Hierarchy in transformers
• AutoTokenizer Automatically Selects the Correct Tokenizer Class
• v5 Separates Tokenizer Architecture from Trained Vocab
• Summary

For experts: If you're already familiar with the concepts and want to understand the changes in v5, go to v5 Separates Tokenizer Architecture from Trained Vocab

Before diving into the changes, let's quickly cover what tokenization does and how the pieces fit together.

What is tokenization?

Language models don't read raw text. They consume sequences of integers usually called token IDs or input IDs. Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground here to visualize tokenization.)

Tokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the transformers and tokenizers libraries.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

text = "Hello world"
tokens = tokenizer(text)

print(tokens["input_ids"])
# [9906, 1917]

print(tokenizer.convert_ids_to_tokens(tokens["input_ids"]))
# ['Hello', 'Ġworld']

This meant: