How I read logs

notesJune 21, 2026 · 5 min read

Reading logs is a skill that nobody teaches directly. You pick it up by being on call, and you pick it up slowly. Here is the routine I settled on after a few years of doing it badly.

1. Find the time first

Before reading anything, decide when the problem started. Ask the person who reported it, look at the graph, check the deploy history. Without a time window you are reading everything, which means you are reading nothing. Narrow it down, then add a few minutes of margin on each side.

2. Look at the first error, not the loudest

When something breaks, it usually breaks noisily. A thousand identical lines appear and drown out everything else. The line that matters is almost always the first unusual one, a few seconds before the flood. Everything after it is a consequence.

# errors in the window, oldest first
journalctl -u myapp --since "14:02" --until "14:10" -p err --no-pager | head -n 40

3. Follow one request all the way through

Pick a single failing request and trace it from start to end. A request id in every log line makes this trivial. If you do not have one, add it this week. It is the cheapest improvement to debugging that I know of.

4. Compare with a healthy moment

Open the logs from an hour before the problem and look at what normal looks like. Half of the lines that seem suspicious turn out to be there every day. Ruling those out early saves a lot of time and a lot of wrong theories.

5. Write down what you ruled out

Keep a plain text file open while you investigate. Note each theory and how you disproved it. It helps when you hand over to someone else, and it helps even more when you return from lunch and no longer remember what you already checked.

Logs do not explain, they only testify. The explanation is yours to build.

A word on log levels

Use fewer levels than you think you need. In practice I want to know three things: something is wrong and needs a person, something is odd but handled, and here is what happened. Everything else is noise that costs money to store and time to skip.