Stop 22 · Interview PreparationTechnical · 2/8

Deep Learning & Architectures

In short

"Why did Transformers replace RNNs?" is a favourite question, and a great one to nail. Show you understand how CNNs, RNNs and attention each see data, and the rest of the round gets easier.

Expect questions on how CNNs, RNNs and Transformers work, why attention replaced recurrence, and practical debugging of training issues (vanishing gradients, overfitting).

The purple blob stands at a blackboard while a row of tiny friends pass a folded note one by one.

Think of it like this

Passing notes vs the blackboard

In class, a message passed bench to bench gets slower and more garbled the further it travels. That's an RNN: one word at a time, memory fading. Write it on the blackboard and every student reads the whole thing at once and focuses on the parts they need. That's attention.

Watch it happen

RNN vs Transformer

RNN / LSTM

Reads one token after another
Hard to train on parallel hardware
Early words fade over long text

Transformer

Every token attends to every other
Whole sequence trains in parallel
Cost grows with length squared

Transformers won because they train in parallel and link distant words directly. The price is attention cost on very long inputs.

The big ideas

What you need to know

CNNs look locally

Small filters slide over an image, finding edges, then shapes, then objects. Weight sharing keeps parameters low.

RNNs carry a memory

Hidden state passes along each step. Gradients can vanish over long sequences; LSTM gates help.

Attention compares everything

Each token scores every other token, then takes a weighted mix. Position encodings add word order back in.

Training troubles to know

Vanishing or exploding gradients, overfitting, bad learning rates. Residual connections and normalisation help deep nets train.

See it in code

Walk through the code

python

1import numpy as np2 3def attention(Q, K, V):4    d_k = Q.shape[-1]5    scores = Q @ K.T / np.sqrt(d_k)                 # how much each token looks at each other6    scores -= scores.max(axis=-1, keepdims=True)    # for numerical stability7    weights = np.exp(scores)8    weights /= weights.sum(axis=-1, keepdims=True)  # softmax: each row sums to 19    return weights @ V10 11tokens = np.random.rand(4, 8)   # 4 tokens, 8 numbers each12out = attention(tokens, tokens, tokens)13print(out.shape)  # (4, 8)

1/3Dot products score every token pair. Dividing by sqrt(d_k) stops scores getting so large the softmax saturates.

Scaled dot-product attention in ten lines. Being able to write this on a whiteboard, and explain why we divide by the square root of d_k, impresses interviewers.

Save yourself the pain

Common mistakes

Oops

Saying "Transformers are better" with no reason.

Do this instead

Give the two reasons: parallel training over the whole sequence, and direct links between distant tokens.

Oops

Forgetting the costs when praising attention.

Do this instead

Mention that attention cost grows with the square of sequence length, which is why long context is expensive.

Your mission

Architecture whiteboard drill

~40 min

Quick check

What's the main reason Transformers train faster than RNNs on modern GPUs?

Go deeper

1Be able to explain backpropagation and attention from first principles, not just at a high level.
2Practice explaining why Transformers largely replaced RNNs for sequence tasks.
3Review common training failure modes and how you'd diagnose them.

More technical at this stop