Stop 17 · MLOps & Model DeploymentTopics · 4/10

Experiment Tracking (MLflow, Weights & Biases)

In short

After 30 training runs, which one had 0.91 accuracy? Was it learning rate 0.01 or 0.001? Experiment tracking writes it all down so you never have to guess again.

Logging hyperparameters, metrics and artifacts across training runs so you can compare experiments systematically instead of relying on memory or scattered notebooks.

The blob in a lab coat writes neatly in a notebook next to a row of colourful test tubes.

Think of it like this

The chemistry practical record book

In chemistry practicals you write every reading: quantity, temperature, colour change, result. Weeks later you can repeat the best one exactly. MLflow and W&B are that record book for training runs: parameters, metrics and the model file, saved for every run.

Watch it happen

Comparing three tracked runs

depth=2

depth=4

depth=8

1/4Run 1 logs max_depth=2 and accuracy 0.89. Shallow trees underfit a little.

The big ideas

What you need to know

Log params

Every setting that can change a result: learning rate, depth, data version, random seed.

Log metrics

Accuracy, loss, F1, per epoch if training is long. Charts show which runs are improving.

Log artifacts

Save the model file, plots and the confusion matrix with the run, so you can download the winner later.

MLflow vs W&B

MLflow is open source and can run locally. W&B is a hosted service with rich dashboards and team features.

See it in code

Walk through the code

python

1import mlflow2from sklearn.datasets import load_iris3from sklearn.ensemble import RandomForestClassifier4from sklearn.model_selection import train_test_split5 6X_tr, X_te, y_tr, y_te = train_test_split(*load_iris(return_X_y=True), random_state=0)7mlflow.set_experiment("iris-rf")8 9for depth in [2, 4, 8]:10    with mlflow.start_run():11        model = RandomForestClassifier(max_depth=depth, random_state=0)12        model.fit(X_tr, y_tr)13        mlflow.log_param("max_depth", depth)14        mlflow.log_metric("accuracy", model.score(X_te, y_te))

1/3Group runs under one named experiment.

Three tracked runs with MLflow. Then run mlflow ui in the same folder and open the local web page to compare them.

Save yourself the pain

Common mistakes

Oops

Logging metrics but not the data version or random seed.

Do this instead

Log anything that changes results. Without the data version, you can't reproduce the best run.

Oops

Picking the best run on the test set after 50 tries.

Do this instead

Compare on a validation set. Keep the test set for one final check, or you overfit to it.

Your mission

Track a sweep

~25 min

Quick check

Two runs have the same accuracy. One logged its data version, one didn't. Which can you trust to reproduce?

Go deeper

1Instrument a training script with MLflow or Weights & Biases logging.
2Compare multiple experiment runs side by side in the tool's dashboard.
3Log enough metadata that a run could be fully reproduced later.