Stop 09 · Natural Language ProcessingTopics · 1/10

Text Preprocessing (Tokenization, Stemming, Lemmatization)

In short

Computers don't read words, they read numbers. Before any text model can learn, raw messy text has to be cleaned and chopped into consistent pieces.

Splitting raw text into usable units and normalizing word forms — the first step in any classic NLP pipeline before embeddings or model input.

The mascot chops vegetables on a cutting board next to bowls of chopped onions, carrots and tomatoes.

Think of it like this

Prepping vegetables before cooking

Before cooking a sabzi you wash, peel and chop everything into similar pieces. Nobody throws a whole muddy cauliflower into the kadhai. Text is the same: lowercase it, strip junk, cut it into tokens and trim words down to their roots.

Watch it happen

From a raw review to clean tokens

Raw text

Clean

Tokenize

Stopwords

Root form

1/5'The delivery boys were running late, but the food was AMAZING!!'

The big ideas

What you need to know

Tokens are the units

A token can be a word, part of a word, or a character. Modern LLMs use subword tokens.

Stemming is rough

Fast rule-based chopping. Results like 'studi' aren't real words, but similar words still match.

Lemmatization is careful

Uses vocabulary and grammar to get the dictionary form. Slower but cleaner.

Don't over-clean

Removing 'not' flips meaning. Emojis and capitals can carry sentiment. Clean to fit your task.

See it in code

Walk through the code

python

1import re2import nltk3from nltk.stem import PorterStemmer, WordNetLemmatizer4 5nltk.download("wordnet", quiet=True)6text = "The delivery boys were running late, but the food was AMAZING!!"7 8tokens = re.findall(r"[a-z]+", text.lower())9stems = [PorterStemmer().stem(t) for t in tokens]10lemmas = [WordNetLemmatizer().lemmatize(t, pos="v") for t in tokens]11print(tokens)12print(stems)13print(lemmas)

1/4Download the WordNet dictionary the lemmatizer needs. It only downloads once.

Stems like 'deliveri' and 'amaz' aren't real words. Lemmas like 'be' and 'run' are.

Save yourself the pain

Common mistakes

Oops

Removing stopwords like 'not' and 'no' before sentiment analysis.

Do this instead

Keep negations. Check what a stopword list contains before using it.

Oops

Stemming text before feeding it to a modern transformer model.

Do this instead

Transformers have their own tokenizer. Give them raw, lightly cleaned text.

Your mission

Clean ten real reviews

~20 min

Quick check

What's the main difference between stemming and lemmatization?

Go deeper

1Practice tokenization, stemming and lemmatization with NLTK or spaCy on real text.
2Learn when stemming's aggressiveness is fine vs. when lemmatization's accuracy matters more.
3Understand that modern LLM tokenizers (BPE) work differently from classic word tokenization.