Stop 11 · Monitoring & ObservabilityTopics · 3/9

Prometheus Fundamentals

In short

Prometheus is the default metrics system in the Kubernetes world. It visits your apps every few seconds, collects their numbers and lets you query and alert on them with one compact language.

The dominant open-source metrics collection and alerting system in the Kubernetes/cloud-native world — it pulls (scrapes) metrics from targets on a schedule and stores them as time-series data.

The mascot walks a lane as a meter reader, noting a wall meter's reading on a clipboard.

Think of it like this

The electricity meter reader

Houses don't phone the electricity board with their readings. The meter reader walks the lane on a schedule, reads each meter and writes the number down with the date. Prometheus works the same way: it pulls readings from each app's /metrics page on a fixed interval.

Watch it happen

One scrape cycle

Prometheus

shop-api

node-exporter

Alertmanager

GET /metrics

1/5Every 15 seconds, Prometheus requests the app's metrics page. The app only has to expose it.

The big ideas

What you need to know

Pull, not push

Prometheus scrapes targets on a schedule. A target that stops answering is itself a useful signal.

Name plus labels

Each series is a metric name plus labels like method and status. Keep labels low in variety.

Four metric types

Counter (only goes up), gauge (up and down), histogram and summary (for latency distributions).

PromQL

rate(http_requests_total[5m]) gives requests per second. Most dashboards and alerts are built from rate().

See it in code

Walk through the code

yaml

1# prometheus.yml2global:3  scrape_interval: 15s4scrape_configs:5  - job_name: shop-api6    static_configs:7      - targets: ["api:3000"]8  - job_name: node9    static_configs:10      - targets: ["node-exporter:9100"]

1/3Scrape every target every 15 seconds unless a job says otherwise.

Then try rate(http_requests_total[5m]) in the Prometheus UI on port 9090. In Kubernetes, the Prometheus Operator finds targets for you.

Save yourself the pain

Common mistakes

Oops

Graphing a raw counter like http_requests_total and seeing a line that only goes up.

Do this instead

Wrap counters in rate() or increase(). The per-second rate is what you actually want to see.

Oops

Adding a label for user ID or full URL with query strings.

Do this instead

That explodes the number of series. Use route templates like /orders/:id and keep IDs out of labels.

Your mission

Scrape your own app

~40 min

Quick check

Why do Prometheus queries usually wrap counters in rate()?

Go deeper

1Set up a local Prometheus instance and configure it to scrape a sample application's metrics.
2Learn the basic PromQL query syntax to retrieve and aggregate a metric.
3Understand why Prometheus's pull model differs from traditional push-based monitoring systems.