Stop 10 · NLP & Modern AITopics · 4/7

Sequence Models for NLP

In short

Translation, chat replies and summaries all turn one sequence of words into another. Sequence models handle input and output of any length, and set the stage for transformers.

Applying RNNs/LSTMs (from Phase 9) specifically to text — processing a sequence of word embeddings to capture context and order, the pre-Transformer standard for tasks like sentiment analysis.

The mascot stands on a railway platform pointing at a long train of colourful carriages.

Think of it like this

Watching the train go by

On the platform you watch carriages pass one by one, in order. By the end, you have a picture of the whole train in your head. An encoder reads words the same way and builds a summary. A decoder then writes the output, one word at a time.

Watch it happen

Encoder-decoder translating 'chai is hot'

English input

Encoder

Decoder

Hindi output

'chai'

1/6The encoder reads the first word and updates its hidden state.

The big ideas

What you need to know

Seq2seq

An encoder reads the input, a decoder generates the output. Lengths can differ, which is what translation needs.

The bottleneck

Squeezing a long sentence into one vector loses detail. Attention was invented to fix exactly this.

Bidirectional reading

For classification, reading left-to-right and right-to-left gives each word context from both sides.

Padding and masks

Batches need equal lengths, so short sentences are padded. Masks tell the model to ignore the padding.

See it in code

Walk through the code

python

1import keras2from keras import layers3 4vectorize = layers.TextVectorization(max_tokens=10000, output_sequence_length=50)5vectorize.adapt(["the food was not good", "loved the service"])   # use your full corpus6 7model = keras.Sequential([8    keras.Input(shape=(1,), dtype="string"),9    vectorize,10    layers.Embedding(10000, 64),11    layers.Bidirectional(layers.LSTM(32)),12    layers.Dense(1, activation="sigmoid"),13])14model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

1/4Build a vocabulary and turn each text into 50 token IDs, padded or cut.

A sequence classifier: raw strings in, a positive or negative probability out. Adapt the vectorizer on your real training text.

Save yourself the pain

Common mistakes

Oops

Setting the sequence length to the longest text in the dataset.

Do this instead

Check the length distribution and pick something like the 95th percentile to save memory.

Oops

Building a seq2seq model from scratch for a real translation product.

Do this instead

Learn the idea here, then use a pretrained transformer for real-world translation or summarisation.

Your mission

Train on IMDB reviews

~40 min

Quick check

In a basic encoder-decoder, what does the decoder receive from the encoder?

Go deeper

1Build a simple LSTM-based text classifier on a small labeled dataset.
2Compare its performance to a simpler Bag of Words / TF-IDF baseline model.
3Learn why sequence order matters for meaning in ways Bag of Words completely ignores.