Table of contents:

  1. Why do we want to train the tokeniser?
  2. How do we train the tokeniser?
  3. What happens to the model?

Why do we want to train the tokeniser?

One of the basic reason why we wanna train the tokeniser is simple, your dataset is very different then what the model was trained on. Basically, there might be words, tokens in your dataset which are very common, but the model haven’t seen much of them while training.

In our use-case, when we are fine-tuning the model for object-detection, the model needs to predict the location type token like <loc0000> to <loc1023>, as the model hasn’t seen these special tokens, it can’t make these token directly, it wold require a combination of token to represent one location token, this is somewhat inefficient.

Training the tokeniser will resolve this inefficiency.

How do we train the tokeniser?

There are basically two way to add new tokens to the existing tokeniser

  1. Extending the vocab size of tokeniser:
  2. Adding new tokens and training the tokeniser back to particular vocab size

Extending the vocab size of tokeniser:

This is what exactly we did in our use-case. This particular way tells the tokeniser in a casual way, like you have these more output tokens which you can output.

from transformers import AutoTokenizer, AutoProcessor

model_id = "HuggingFaceTB/SmolVLM-256M-Instruct" # "google/gemma-3-4b-pt"
# Load processor and tokenizer
processor = AutoProcessor.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Get original sizes
original_vocab_size = tokenizer.vocab_size
original_total_size = len(tokenizer)

print(f"Original vocab size (pretrained): {original_vocab_size}")
print(f"Original total tokenizer size (includes added tokens): {original_total_size}")

# Add new location tokens
location_tokens = [f"<loc{i:04}>" for i in range(1024)]
added_tokens_count = tokenizer.add_tokens(location_tokens, special_tokens=True)

# Get updated sizes
new_total_size = len(tokenizer)

print(f"Number of new tokens added: {added_tokens_count}")
print(f"New total tokenizer size: {new_total_size}")
print(f"New vocab size (still unchanged): {tokenizer.vocab_size}")  # This remains same

# Attach updated tokenizer to processor if needed
processor.tokenizer = tokenizer