Catalog My learning Blog
Linux Command Line Tutorial for Developers · Phase 4 · Lesson 24 · 3 min read

Linux Logs and journalctl Tutorial

On this page
  1. What the systemd journal collects
  2. journalctl filters you'll use daily
  3. The journalctl priority trap
  4. Limiting journal size
  5. Text log files in /var/log and logrotate
  6. Check yourself

What the systemd journal collects

On systemd machines, the journal captures everything a service writes to stdout and stderr, plus kernel messages and system logs, with metadata (unit, PID, priority, timestamps). Many systems also write text files under /var/log/ (nginx access logs, auth.log), but the journal is usually the first place to look.

journalctl filters you'll use daily

CommandShows
journalctl -u myappOne service
journalctl -u myapp -fFollow live
journalctl -u myapp --since "1 hour ago"A time window (also --since yesterday, --until "2026-10-14 23:00")
journalctl -u myapp -bSince the last boot (-b -1 the boot before)
journalctl -u myapp -n 50Last 50 lines
journalctl -p errPriority err and worse
journalctl -g 'timed out'Lines matching a pattern
journalctl -kKernel messages (OOM kills, disk errors)
journalctl -o jsonStructured output with all fields

Combine them: journalctl -u myapp -p warning --since today.

From the crashing service in Day 19:

Shell
journalctl -u myapp --since "-1min" -o short
Code
lima-lab python3[10233]: ERROR: connection to database timed out
lima-lab systemd[1]: myapp.service: Failed with result 'exit-code'.
lima-lab systemd[1]: myapp.service: Scheduled restart job, restart counter is at 1.
lima-lab python3[10238]: ERROR: connection to database timed out

The application's own message and systemd's view of the failure, interleaved in order.

Services write to stdout and stderr, the kernel and system tools log too; the journal stores every entry with metadata, and journalctl filters by unit, time, priority and pattern.
Services write to stdout and stderr, the kernel and system tools log too; the journal stores every entry with metadata, and journalctl filters by unit, time, priority and pattern.

The journalctl priority trap

Filtering by priority seems the obvious way to find errors:

Shell
journalctl -u myapp -p err --since "-5min"

It returned nothing, even though the service had just logged ERROR: connection to database timed out several times. The JSON output explains why:

Code
priority of the ERROR lines: ['6'] (6 = info)

Everything a service prints to stdout or stderr is logged at info priority by default. The word "ERROR" in the text means nothing to the journal. Two fixes:

  • Prefix lines with a syslog level: systemd recognizes <3> (err), <4> (warning) and so on at the start of a line. After changing the app to print <3>ERROR: connection to database timed out, journalctl -p err found it.
  • Log through a library that writes to the journal with proper priorities (systemd's native protocol), or set SyslogLevel= in the unit to change the default priority of its output, or log structured JSON to stdout and let your log pipeline parse levels.

Until then, search by text: journalctl -u myapp -g ERROR.

Limiting journal size

  • Persistence: if /var/log/journal/ exists, logs survive reboots; otherwise they live in memory under /run.
  • Size: journalctl --disk-usage; cap it with SystemMaxUse=1G in /etc/systemd/journald.conf, or clean up with journalctl --vacuum-time=14d.
  • Rate limits: journald drops messages from services that log too fast, noting "Suppressed N messages".
  • Central logging: ship logs off the machine (Vector, Fluent Bit, the OpenTelemetry Collector, or systemd-journal-upload), so they survive the server and can be searched across many machines (see the observability course).

Text log files in /var/log and logrotate

For files under /var/log/, use the tools from earlier days: tail -F, less +F, grep, awk. logrotate rotates and compresses them; make sure services reopen their files after rotation (Day 14 shows what happens when they don't).

Check yourself

Output to stdout and stderr is logged at info priority by default; the word ERROR in the text doesn't set the priority.

journalctl -k shows kernel messages from the journal, and dmesg reads the kernel ring buffer directly. Search for Out of memory or oom-kill to find which process was killed.

journalctl -u service -b. Add -f to follow new lines, --since "1 hour ago" to narrow by time, or -b -1 for the previous boot.

Set SystemMaxUse in /etc/systemd/journald.conf and restart journald for a permanent limit, or run journalctl --vacuum-size=500M to shrink it immediately. Without a limit, the journal can use up to 10 percent of the filesystem.

It rotates, compresses and deletes old text log files on a schedule, with options to signal services to reopen their logs.