Stop 08 · Deep Learning ArchitecturesTopics · 4/11

LSTMs & GRUs

In short

Plain RNNs forget what they read a few dozen steps back. LSTMs and GRUs add gates that decide what to keep, what to forget and what to share, so useful information survives much longer.

Gated variants of RNNs that use memory cells and gates to retain information over longer sequences — solved the vanishing gradient problem that limited plain RNNs.

The blob holds an eraser in one hand and a highlighter in the other over an open notebook.

Think of it like this

A topper's revision notebook

Before exams, a topper's notebook is managed carefully. Rub out what's no longer needed, highlight new points worth keeping, and before each test pull out just the bits that matter today. LSTM gates are that eraser, highlighter and quick-revision page.

Watch it happen

One LSTM step

Forget gate

Input gate

Cell update

Output gate

1/4Look at the new word and the last hidden state. Output a 0 to 1 value per memory slot: how much to keep.

The big ideas

What you need to know

A separate cell state

LSTMs carry a cell state alongside the hidden state. It's updated mostly by addition, which helps memory last.

Gates are soft switches

Each gate is a sigmoid layer giving values between 0 (block) and 1 (let through) for every memory slot.

GRU: a lighter cousin

A GRU merges forget and input into one update gate and has no separate cell. Fewer weights, often similar accuracy.

Still useful

Transformers lead in text, but LSTMs and GRUs remain handy for small time series and on-device models.

See it in code

Walk through the code

python

1import torch2import torch.nn as nn3 4x = torch.randn(4, 50, 32)   # 4 sequences, 50 steps, 32 features5 6lstm = nn.LSTM(32, 64, batch_first=True)7out, (h, c) = lstm(x)        # LSTM keeps a hidden state AND a cell state8 9gru = nn.GRU(32, 64, batch_first=True)10out_g, h_g = gru(x)          # GRU merges them into one hidden state11 12count = lambda m: sum(p.numel() for p in m.parameters())13print(count(lstm), count(gru))   # GRU has 3/4 of the LSTM's weights

1/3An LSTM returns outputs plus a tuple of final hidden state h and final cell state c.

Same input, two gated cells. The LSTM returns an extra cell state, and the GRU is smaller.

Save yourself the pain

Common mistakes

Oops

Unpacking an LSTM like an RNN: out, h = lstm(x).

Do this instead

An LSTM returns out, (h, c). Unpack the tuple or you'll pass a tuple where a tensor is expected.

Oops

Assuming LSTMs remember everything forever.

Do this instead

They remember far longer than plain RNNs, but very long documents still lose detail. Attention helps there.

Your mission

RNN vs LSTM vs GRU

~40 min

Quick check

What is the job of the forget gate in an LSTM?

Go deeper

1Train an LSTM on a sequence task and compare it to a plain RNN.
2Learn what the forget, input and output gates each control.
3Understand why GRUs are a simpler, often faster alternative to LSTMs.