One of the basic reason why we wanna train the tokeniser is simple, your dataset is very different then what the model was trained on. Basically, there might be words, tokens in your dataset which are very common, but the model haven’t seen much of them while training.
In our use-case, when we are fine-tuning the model for object-detection, the model needs to predict the location type token like <loc0000> to <loc1023>, as the model hasn’t seen these special tokens, it can’t make these token directly, it wold require a combination of token to represent one location token, this is somewhat inefficient.
Training the tokeniser will resolve this inefficiency.
There are basically two way to add new tokens to the existing tokeniser
This is what exactly we did in our use-case. This particular way tells the tokeniser in a casual way, like you have these more output tokens which you can output.
from transformers import AutoTokenizer, AutoProcessor
model_id = "HuggingFaceTB/SmolVLM-256M-Instruct" # "google/gemma-3-4b-pt"
# Load processor and tokenizer
processor = AutoProcessor.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Get original sizes
original_vocab_size = tokenizer.vocab_size
original_total_size = len(tokenizer)
print(f"Original vocab size (pretrained): {original_vocab_size}")
print(f"Original total tokenizer size (includes added tokens): {original_total_size}")
# Add new location tokens
location_tokens = [f"<loc{i:04}>" for i in range(1024)]
added_tokens_count = tokenizer.add_tokens(location_tokens, special_tokens=True)
# Get updated sizes
new_total_size = len(tokenizer)
print(f"Number of new tokens added: {added_tokens_count}")
print(f"New total tokenizer size: {new_total_size}")
print(f"New vocab size (still unchanged): {tokenizer.vocab_size}") # This remains same
# Attach updated tokenizer to processor if needed
processor.tokenizer = tokenizer