Stop 17 · MLOps & Model DeploymentTopics · 9/10

Kubernetes for ML (Overview)

In short

One container on one server is fine for a demo. When traffic spikes, a server dies or you need GPUs, Kubernetes runs many copies and keeps them healthy for you.

Orchestrating containerized model-serving workloads at scale — handles autoscaling, load balancing and self-healing for production ML services.

The blob in a station master's cap waves travellers toward a row of open ticket windows.

Think of it like this

Ticket counters at a big station

During Diwali rush, the station opens more ticket windows. If one window's printer breaks, its queue is sent to another. The station manager doesn't care which clerk serves you, only that enough windows are open. Kubernetes is that manager for your model containers.

Watch it happen

Where your model runs in a cluster

Cluster

CPU node

GPU node

api pod 1

api pod 2

llm pod

1/5A cluster is a group of machines that Kubernetes manages as one big computer.

The big ideas

What you need to know

Pods run containers

A pod is the smallest unit, usually one container. Deployments keep a set number of identical pods running.

Autoscaling

The Horizontal Pod Autoscaler adds pods when CPU or custom metrics like queue length go up.

Ask for resources

Each container declares the CPU, memory or GPU it needs, so Kubernetes can place it on the right node.

ML add-ons

Tools like KServe and Kubeflow add model serving and pipelines on top of Kubernetes.

See it in code

Walk through the code

yaml

1apiVersion: apps/v12kind: Deployment3metadata:4  name: spam-model5spec:6  replicas: 37  selector:8    matchLabels: { app: spam-model }9  template:10    metadata:11      labels: { app: spam-model }12    spec:13      containers:14        - name: api15          image: ghcr.io/you/spam-model:1.2.016          resources: { requests: { cpu: "500m", memory: "1Gi" } }

1/4Always keep 3 pods. If one dies, Kubernetes starts another.

A Deployment that keeps three copies of your model container running. Apply it with kubectl apply -f deploy.yaml.

Save yourself the pain

Common mistakes

Oops

Reaching for Kubernetes for a college project with ten users.

Do this instead

A single container on a simple host is enough. Kubernetes pays off with scale and a team to run it.

Oops

Using the latest image tag in production.

Do this instead

Pin a version tag so rollbacks are easy and you know what's running.

Your mission

Run a local cluster

~40 min

Quick check

One of your three model pods crashes at 2 am. What does Kubernetes do?

Go deeper

1Read Kubernetes core concepts: pods, deployments, services, autoscaling.
2Deploy a containerized model-serving app to a local Kubernetes cluster (minikube/kind).
3Understand when a project actually needs Kubernetes vs. simpler managed hosting.