Stop 08 · Deep Learning ArchitecturesTopics · 10/11

Autoencoders

In short

An autoencoder learns to copy its input through a narrow bottleneck. Copying sounds pointless, but the bottleneck forces it to learn what really matters in the data.

Networks trained to compress input into a smaller representation and reconstruct it — used for dimensionality reduction, denoising, and as a building block for generative models.

The blob squeezes a thick book into a tiny flashcard, then unfolds the flashcard back into a book.

Think of it like this

One flashcard per chapter

Your teacher says: squeeze the whole chapter onto one flashcard, then rewrite the chapter from only that card. You can't copy everything, so you keep the key ideas. The flashcard is the bottleneck, and what's left out was noise.

Watch it happen

Squeeze, then rebuild

Input 784

Encoder

Code 32

Decoder

Output 784

Compare

1/6A 28 × 28 digit image, flattened to 784 pixel values.

The big ideas

What you need to know

The bottleneck is the point

If the middle were as wide as the input, the network could just copy. A narrow code forces it to summarise.

Anomaly detection

Train on normal data only. Unusual inputs reconstruct badly, so a high error flags possible fraud or faults.

Denoising

Feed noisy images in and ask for clean ones out. The network learns to remove the noise.

VAEs can generate

Variational autoencoders shape the latent space so you can sample new codes and decode brand-new images.

See it in code

Walk through the code

python

1import torch2import torch.nn as nn3 4class AutoEncoder(nn.Module):5    def __init__(self):6        super().__init__()7        self.encoder = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 32))8        self.decoder = nn.Sequential(nn.Linear(32, 128), nn.ReLU(), nn.Linear(128, 784), nn.Sigmoid())9 10    def forward(self, x):11        code = self.encoder(x)          # squeeze 784 pixels into 32 numbers12        return self.decoder(code)       # rebuild 784 pixels from those 3213 14model = AutoEncoder()15x = torch.rand(16, 784)                 # stand-in for 16 flattened MNIST images16loss = nn.functional.mse_loss(model(x), x)   # the target is the input itself

1/4The encoder narrows 784 values down to a 32-number code.

The only unusual line is the last one: the input is also the target. Train it like any other PyTorch model.

Save yourself the pain

Common mistakes

Oops

Making the bottleneck as wide as the input.

Do this instead

Then the network learns to copy and nothing useful. Keep the code much smaller than the input.

Oops

Expecting a plain autoencoder to generate realistic new images from random codes.

Do this instead

Its latent space has gaps. Use a VAE or another generative model if you want to sample new data.

Your mission

Compress MNIST

~40 min

Quick check

How can an autoencoder trained only on normal transactions help catch fraud?

Go deeper

1Build a simple autoencoder for image reconstruction or dimensionality reduction.
2Learn the difference between a standard autoencoder and a variational autoencoder (VAE).
3Use the learned latent space to visualize how the model compresses data.