Stop 10 · NLP & Modern AITopics · 1/7

Text Preprocessing (Tokenization, Stemming, Lemmatization)

In short

Computers don't read words, they read numbers. Before any text model can learn, raw messy text has to be cleaned and chopped into consistent pieces.

Turning raw text into a cleaner, more analyzable form — tokenization splits text into words/subwords, stemming crudely chops word endings, lemmatization properly reduces words to their dictionary root.

The mascot chops vegetables on a cutting board next to bowls of chopped onions, carrots and tomatoes.

Think of it like this

Prepping vegetables before cooking

Before cooking a sabzi you wash, peel and chop everything into similar pieces. Nobody throws a whole muddy cauliflower into the kadhai. Text is the same: lowercase it, strip junk, cut it into tokens and trim words down to their roots.

Watch it happen

From a raw review to clean tokens

Raw text

Clean

Tokenize

Stopwords

Root form

1/5'The delivery boys were running late, but the food was AMAZING!!'

The big ideas

What you need to know

Tokens are the units

A token can be a word, part of a word, or a character. Modern LLMs use subword tokens.

Stemming is rough

Fast rule-based chopping. Results like 'studi' aren't real words, but similar words still match.

Lemmatization is careful

Uses vocabulary and grammar to get the dictionary form. Slower but cleaner.

Don't over-clean

Removing 'not' flips meaning. Emojis and capitals can carry sentiment. Clean to fit your task.

See it in code

Walk through the code

python

1import re2import nltk3from nltk.stem import PorterStemmer, WordNetLemmatizer4 5nltk.download("wordnet", quiet=True)6text = "The delivery boys were running late, but the food was AMAZING!!"7 8tokens = re.findall(r"[a-z]+", text.lower())9stems = [PorterStemmer().stem(t) for t in tokens]10lemmas = [WordNetLemmatizer().lemmatize(t, pos="v") for t in tokens]11print(tokens)12print(stems)13print(lemmas)

1/4Download the WordNet dictionary the lemmatizer needs. It only downloads once.

Stems like 'deliveri' and 'amaz' aren't real words. Lemmas like 'be' and 'run' are.

Save yourself the pain

Common mistakes

Oops

Removing stopwords like 'not' and 'no' before sentiment analysis.

Do this instead

Keep negations. Check what a stopword list contains before using it.

Oops

Stemming text before feeding it to a modern transformer model.

Do this instead

Transformers have their own tokenizer. Give them raw, lightly cleaned text.

Your mission

Clean ten real reviews

~20 min

Quick check

What's the main difference between stemming and lemmatization?

Go deeper

1Tokenize a sample paragraph and inspect the resulting tokens.
2Compare stemming and lemmatization on the same set of words and note the differences.
3Learn when heavy preprocessing helps (classic ML) versus when it can hurt (modern transformer models).