Stop 19 · Monitoring & LoggingTopics · 4/9

Centralized Logging

In short

With five servers, SSHing into each one to grep logs is misery. Centralized logging pulls every line into one searchable place, so you find the problem in seconds.

Shipping logs from all your servers/containers to one searchable place (e.g. Datadog, ELK stack, CloudWatch Logs) instead of SSH-ing into individual machines to read log files.

The purple blob sits in a control room in front of a wall of small monitors, pointing at one.

Think of it like this

The CCTV control room

A mall has cameras on every floor. Nobody walks around checking each one. All feeds come to one control room, where a guard can rewind any corridor at any time. Centralized logging is that control room for your servers.

Watch it happen

Where a log line travels

Apps

Shipper

Store

Alert

1/5Each app writes JSON logs to stdout. Containers and VMs keep them locally for only a short time.

The big ideas

What you need to know

Servers are temporary

Containers die and autoscaled VMs vanish, taking local logs with them. Ship logs off the box quickly.

Consistent fields

Every service should use the same field names, like service, env, level and reqId, so one query works everywhere.

Retention costs money

Keep detailed logs for days or weeks, not forever. Decide retention per level and environment.

Logs are sensitive

Access to logs is access to user data. Restrict who can search them and redact before shipping.

See it in code

Walk through the code

yaml

1# vector.yaml: read Docker logs, parse JSON, ship to Loki2sources:3  docker: { type: docker_logs }4transforms:5  parse:6    type: remap7    inputs: [docker]8    source: '. = merge(., object!(parse_json!(.message)))'9sinks:10  loki:11    type: loki12    inputs: [parse]13    endpoint: http://loki:310014    labels: { service: "{{ container_name }}" }15    encoding: { codec: json }

1/3Read the stdout of every container on this host.

Vector is one of several shippers. The shape is always the same: a source, optional transforms, and a sink.

Save yourself the pain

Common mistakes

Oops

Each service logging plain text in its own style.

Do this instead

Searching across services then needs a different pattern per service. Agree on JSON and shared field names.

Oops

Shipping debug-level logs from production and keeping them forever.

Do this instead

Storage bills climb fast. Ship info and above, set retention, and drop noisy health check logs.

Your mission

One place for all logs

~45 min

Quick check

Your autoscaler removed a server at 3:05. At 3:20 you need its logs from 3:00. Where are they?

Go deeper

1Ship a Node app's logs to a centralized logging service, even a free-tier one.
2Practice searching and filtering logs by request ID or error type.
3Understand why this becomes essential once you run more than one server instance.