Stop 09 · Deep Learning FundamentalsTopics · 7/10

Recurrent Neural Networks (RNNs) & LSTMs

In short

In 'the food was not good', the word 'not' flips everything. RNNs read sequences step by step and carry a memory forward, so order matters to them.

Architectures designed for sequential data (text, time series) that maintain a 'memory' of previous inputs — LSTMs specifically solve RNNs' tendency to forget long-range dependencies.

The mascot listens to a small radio while writing notes in a pocket diary.

Think of it like this

Following the cricket commentary

Listening to a match on the radio, you don't remember every ball. You keep a running picture: score, wickets, who's batting. An LSTM is like also keeping a small diary of the key moments, so a wicket from 20 overs ago still counts.

Watch it happen

An LSTM reading 'not good' one word at a time

Next word

LSTM cell

Cell memory

Output

x1 = 'the'

1/7The first word arrives as a vector. The cell mixes it with an empty hidden state.

The big ideas

What you need to know

Same cell, every step

An RNN applies the same weights at each time step, passing a hidden state forward like a running summary.

Plain RNNs forget

Gradients shrink over long sequences, so simple RNNs struggle to link things far apart.

LSTM gates

Forget, input and output gates decide what to drop, store and reveal. GRUs are a lighter version.

Now often replaced

For text, transformers mostly took over. LSTMs remain handy for smaller time-series and sensor problems.

See it in code

Walk through the code

python

1import keras2from keras import layers3 4model = keras.Sequential([5    keras.Input(shape=(30, 1)),          # 30 time steps, 1 value each6    layers.LSTM(32),7    layers.Dense(1),                     # predict the next value8])9model.compile(optimizer="adam", loss="mse")10model.summary()

1/3Each sample is a sequence of 30 steps with one number per step.

An LSTM that reads the last 30 readings, like daily temperatures, and predicts the next one.

Save yourself the pain

Common mistakes

Oops

Shuffling time steps inside a sequence before training.

Do this instead

Keep each sequence in order. Shuffle whole samples between batches, never the steps within one.

Oops

Passing a 2D array (samples, steps) to an LSTM layer.

Do this instead

LSTMs expect 3D input: (samples, time steps, features). Add the last axis with reshape or np.expand_dims.

Your mission

Predict a sine wave

~30 min

Quick check

What problem do LSTMs solve compared to plain RNNs?

Go deeper

1Read a conceptual explanation of why sequence order matters for these architectures.
2Build a simple LSTM for a basic sequence prediction task.
3Learn why Transformers (Phase 10) have largely replaced RNNs/LSTMs for most NLP tasks today.