Stop 07 · Deep Learning FoundationsTopics · 11/11

GPUs & CUDA Basics

In short

Training a model on a CPU can take days. The same job on a GPU can take hours. Knowing how to move work onto a GPU, and why it helps, is everyday AI engineering.

GPUs parallelize the matrix operations behind neural networks, cutting training time from days to hours — understanding CUDA basics helps you debug device and memory issues.

The blob stands at the front of a big exam hall where rows of small students all write at once.

Think of it like this

One professor vs a full exam hall

Give a professor a thousand simple sums and she'll do them one by one, very fast. Give the same sums to an exam hall of a thousand students, one each, and they're done in one go. A CPU is the professor. A GPU is the exam hall.

Watch it happen

Running a layer on the GPU

CPU + RAM

GPU memory

GPU cores

model.to('cuda')

1/5The model's weights are copied from normal RAM into the GPU's own memory.

The big ideas

What you need to know

Deep learning is matrix maths

Layers are mostly big matrix multiplications, which split into many small independent sums. That's perfect for parallel cores.

CUDA is NVIDIA's toolkit

CUDA lets programs run on NVIDIA GPUs. PyTorch and TensorFlow call CUDA libraries for you, so you rarely write kernels yourself.

VRAM is the limit

Model weights, activations and optimiser state all need GPU memory. 'CUDA out of memory' means lower the batch size or model size.

Copying costs time

Moving data between RAM and GPU is slow compared to compute. Keep tensors on the GPU and avoid moving them back and forth.

See it in code

Walk through the code

python

1import torch2 3device = "cuda" if torch.cuda.is_available() else "cpu"4print("using", device)5 6model = torch.nn.Linear(1024, 1024).to(device)   # weights move to GPU memory7x = torch.randn(4096, 1024, device=device)       # data created on the GPU8 9y = model(x)            # runs as GPU kernels across thousands of cores10print(y.device)         # cuda:0 on a GPU machine11result = y.cpu()        # copy back to RAM before using NumPy