Stop 08 · Deep Learning ArchitecturesTopics · 5/11

Sequence-to-Sequence Models

In short

Translation, summarisation and speech-to-text all take one sequence in and give a different-length sequence out. Seq2seq models were built for exactly this, and their encoder-decoder idea lives on in transformers.

Encoder-decoder architectures that map one sequence to another (e.g. translation) — the architectural predecessor to the Transformer and still conceptually relevant.

The blob wears a headset in an interpreter's booth, listening on one side and speaking on the other.

Think of it like this

The conference interpreter

An interpreter listens to a full sentence in Tamil, holds its meaning in mind, then speaks it in Hindi word by word, using each word she's said to choose the next. Seq2seq has the same two halves: an encoder that listens and a decoder that speaks.

Watch it happen

Translating one sentence

Encoder

Context

Decoder

read 'how are you'

1/6The encoder reads the source sentence one token at a time, updating its hidden state.

The big ideas

What you need to know

Encoder reads

The encoder turns the input sequence into hidden states. Early seq2seq kept only the final one as the summary.

Decoder writes

The decoder generates output tokens one by one, each conditioned on the context and the tokens so far.

Teacher forcing

During training, the decoder is fed the correct previous word, not its own guess. It speeds up learning a lot.

The bottleneck

Squeezing a long sentence into one vector loses detail. Attention fixed this by letting the decoder look back at every input step.

See it in code

Walk through the code

python

1import torch, torch.nn as nn2 3V, H = 1000, 128                      # vocab size, hidden size4emb = nn.Embedding(V, H)5encoder = nn.GRU(H, H, batch_first=True)6decoder = nn.GRU(H, H, batch_first=True)7to_vocab = nn.Linear(H, V)8 9src = torch.randint(0, V, (1, 7))     # one source sentence, 7 token ids10_, context = encoder(emb(src))        # whole sentence squeezed into one vector11 12token, h, result = torch.tensor([[1]]), context, []   # 1 = <start>13for _ in range(10):                   # greedy decoding, one word at a time14    out, h = decoder(emb(token), h)15    token = to_vocab(out).argmax(-1)  # pick the most likely next word16    result.append(token.item())

1/3A shared embedding table, a GRU encoder, a GRU decoder and a layer that scores every vocabulary word.

An untrained skeleton, so the output is random ids. The shape of the loop is what matters: encode once, decode step by step.

Save yourself the pain

Common mistakes

Oops

Training the decoder only on its own predictions from day one.

Do this instead

Use teacher forcing early on, feeding the true previous token, then gradually mix in the model's own guesses.

Oops

Always picking the single most likely word at each step.

Do this instead

Greedy decoding can paint itself into a corner. Beam search keeps a few candidate sentences and picks the best overall.

Your mission

Teach a model to reverse

~45 min

Quick check

Why did early seq2seq models struggle with long sentences?

Go deeper

1Learn the encoder-decoder pattern at a conceptual level with a translation example.
2Understand the information bottleneck problem that motivated attention.
3Trace how seq2seq models evolved into attention-based and then Transformer models.