Stop 08 · Deep Learning ArchitecturesTopics · 3/11

Recurrent Neural Networks (RNNs)

In short

Words, stock prices and sensor readings come in order, and order changes meaning. RNNs read one step at a time and carry a memory forward, which made them the first big tool for sequences.

Networks with loops that carry a hidden state across time steps, designed for sequential data like text and time series — the precursor to LSTMs and Transformers.

The blob in headphones commentates into a microphone while watching a cricket match from a booth.

Think of it like this

The radio cricket commentator

A commentator doesn't restart with every ball. He remembers the score, who's bowling and the pressure so far, and each new ball updates that picture. An RNN does the same: each new word updates a running memory called the hidden state.

Watch it happen

Reading 'the food was not bad'

the

food

was

not

bad

Output

1/6Start with an empty memory h0. Read 'the' and update the memory to h1.

The big ideas

What you need to know

One cell, reused

The same weights process every time step. That's why an RNN can read sequences of any length.

Hidden state is the memory

At each step, h_new = tanh(W·x + U·h_old + b). The new word and the old memory both shape it.

Memory fades

Gradients shrink as they flow back through many steps, so plain RNNs struggle to remember things from far back.

Slow, step by step

Step 10 must wait for step 9. That's hard to parallelise, one reason transformers replaced RNNs for most text work.

See it in code

Walk through the code

python

1import torch2import torch.nn as nn3 4rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True)5 6x = torch.randn(4, 10, 8)    # 4 sentences, 10 words each, 8 numbers per word7outputs, h_last = rnn(x)8 9print(outputs.shape)  # (4, 10, 16): a hidden state after every word10print(h_last.shape)   # (1, 4, 16): the final memory for each sentence

1/3Each word comes in as 8 numbers. The memory is 16 numbers wide.

An RNN returns the memory after every step and the final memory. Classifiers usually use the final one.

Save yourself the pain

Common mistakes

Oops

Forgetting batch_first=True and feeding (batch, time, features).

Do this instead

PyTorch RNNs default to (time, batch, features). Set batch_first=True or transpose your input.

Oops

Using a plain RNN for long documents.

Do this instead

Plain RNNs forget long-range context. Use an LSTM or GRU, or a transformer for most text tasks today.

Your mission

Predict the next number

~35 min

Quick check

What does an RNN pass from one time step to the next?

Go deeper

1Implement a simple RNN for a sequence prediction task.
2Learn why plain RNNs struggle with long sequences (vanishing gradients).
3Understand the conceptual link between RNNs and how Transformers eventually replaced them.