Stop 08 · Deep Learning ArchitecturesTopics · 8/11

Positional Encoding

In short

Attention treats a sentence like a bag of words. 'Dog bites man' and 'man bites dog' would look identical. Positional encoding is the small trick that tells a transformer who came first.

A technique that injects information about token order into a Transformer, since self-attention alone has no sense of sequence position.

The blob stamps a small round seal on the corner of each loose paper sheet in a scattered pile.

Think of it like this

Numbering your answer sheets

You hand in ten loose sheets with no page numbers, and they get shuffled on the examiner's desk. Your answers are all there, but the order is lost. Writing a number on every page fixes it. Positional encoding stamps a position onto every token.

Watch it happen

Same words, different order

dog

bites

man

1/4Without positions, attention sees just the set {dog, bites, man}. Who bit whom is unclear.

The big ideas

What you need to know

Added to embeddings

A position vector of the same size is added to each token's embedding before the first transformer block.

Sinusoidal: waves of many speeds

The original transformer used sine and cosine waves at different frequencies. Each position gets a unique pattern.

Learnt positions

BERT and GPT-2 instead learn one vector per position, which caps how long inputs can be.

Rotary (RoPE)

Many recent LLMs rotate query and key vectors by an angle based on position, so attention sees relative distance directly.

See it in code

Walk through the code

python

1import torch2 3def sinusoidal_positions(seq_len, d_model):4    pos = torch.arange(seq_len).unsqueeze(1)                 # 0, 1, 2, ...5    i = torch.arange(0, d_model, 2)                          # even dimensions6    angle = pos / (10000 ** (i / d_model))7    pe = torch.zeros(seq_len, d_model)8    pe[:, 0::2] = torch.sin(angle)                           # even dims: sine9    pe[:, 1::2] = torch.cos(angle)                           # odd dims: cosine10    return pe11 12tokens = torch.randn(10, 16)              # 10 token embeddings, 16 dims each13x = tokens + sinusoidal_positions(10, 16) # same words, now with a sense of order14print(x.shape)

1/3Each position is divided by a different wavelength per dimension pair, from fast to very slow.

The formula from the original transformer paper. Low dimensions wiggle fast, high dimensions change slowly.

Save yourself the pain

Common mistakes

Oops

Concatenating position vectors and then wondering why shapes don't match.

Do this instead

Standard transformers add positions element-wise, so the position vector must have the same size as the embedding.

Oops

Assuming a model with learnt positions works on longer inputs than it trained on.

Do this instead

Learnt position tables have a fixed length. Beyond it there's simply no vector for those positions.

Your mission

See the waves

~20 min

Quick check

Why does a transformer need positional encoding at all?

Go deeper

1Learn the sinusoidal positional encoding formula and why it was chosen.
2Understand learned vs. fixed positional encodings and where each is used.
3Read about rotary positional embeddings (RoPE) used in modern LLMs.