Stop 09 · Natural Language ProcessingTopics · 7/10

Pretrained NLP Models (BERT, RoBERTa)

In short

Training a language model from zero needs huge data and money. BERT-style models have already read a mountain of text. You borrow that knowledge and fine-tune it for your task in an afternoon.

Encoder-only transformer models pretrained on massive text corpora — the standard foundation for fine-tuning classification, NER and similar 'understanding' tasks.

The blob happily fills in blanks in a huge stack of workbooks with a pencil.

Think of it like this

Years of fill-in-the-blanks

Remember English workbooks full of 'fill in the blanks'? Do thousands of them and you start to really understand sentences. BERT was trained the same way: hide a word, guess it, repeat across billions of words. Then it's ready for real jobs.

Watch it happen

Masked language modelling

drank

hot

[MASK]

the

station

1/4During pretraining, about 15% of tokens are hidden. Here the model must guess the masked word.

The big ideas

What you need to know

Bidirectional

BERT sees left and right context together. That suits understanding tasks, but it isn't built to generate long text.

RoBERTa: BERT, trained better

RoBERTa dropped next-sentence prediction, used dynamic masking and trained on much more data for longer.

Fine-tuning is the point

Add a small task head on top and train for a few epochs on your labelled data. Most of the knowledge is already there.

Know the limits

Base BERT and RoBERTa read at most 512 tokens. Longer documents need chunking or a long-context model.

See it in code

Walk through the code

python

1from transformers import pipeline2 3bert = pipeline("fill-mask", model="google-bert/bert-base-uncased")4for guess in bert("I drank a hot cup of [MASK] at the station.")[:3]:5    print("BERT:", guess["token_str"], round(guess["score"], 2))6 7roberta = pipeline("fill-mask", model="FacebookAI/roberta-base")8for guess in roberta("I drank a hot cup of <mask> at the station.")[:3]:9    print("RoBERTa:", guess["token_str"], round(guess["score"], 2))

1/2BERT's mask token is [MASK]. The pipeline returns the top guesses with scores.

Ask both models to fill the blank. Notice they use different mask tokens because their tokenizers differ.

Save yourself the pain

Common mistakes

Oops

Using BERT to write long answers or chat replies.

Do this instead

BERT is an encoder for understanding. For generating text, use a decoder model like a GPT-style LLM or an encoder-decoder like T5.

Oops

Picking an uncased model when capital letters matter, like in NER.

Do this instead

Use a cased model when 'Apple' vs 'apple' carries meaning. Uncased models lowercase everything first.

Your mission

Probe what BERT knows

~20 min

Quick check

What task was BERT mainly pretrained on?

Go deeper

1Fine-tune a BERT-family model on a text classification task.
2Learn the difference between BERT's masked-language-model pretraining and GPT's next-token prediction.
3Understand why encoder-only models are typically used for understanding tasks, not generation.