Catalog My learning Blog
Linux Command Line Tutorial for Developers · Phase 4 · Lesson 28 · 4 min read

Troubleshooting a Production Linux Server

On this page
  1. The incident: checkout returns 503
  2. 1. Confirm and characterize the symptom
  3. 2. Check the services with systemctl
  4. 3. Read the logs with journalctl
  5. 4. Check the disk with df and du
  6. 5. Fix and verify
  7. Why it didn't fail sooner
  8. A general method for troubleshooting Linux servers
  9. Check yourself

The incident: checkout returns 503

Checkout pages return errors. On the server, systemctl status says both nginx and the app are active (running). Nothing has been deployed today. This walkthrough was reproduced on the course's Ubuntu VM: nginx in front of the app from Day 19, with the app's data on its own small filesystem.

1. Confirm and characterize the symptom

Code
for i in 1 2 3; do curl -s -o /dev/null -w '%{http_code} ' http://127.0.0.1/checkout; done
Code
503 503 503

Consistent, not intermittent. Every request fails.

2. Check the services with systemctl

Shell
systemctl is-active myapp nginx
Code
active
active

"Active" only means the process is running, not that it works. Health checks that make real requests (Day 26) catch what status checks miss.

3. Read the logs with journalctl

Shell
journalctl -u myapp --since "-10min" -o cat | tail -5
Code
ERROR writing session: [Errno 28] No space left on device

Errno 28 is ENOSPC. The app fails to write session data and returns 500, which nginx turns into 503.

4. Check the disk with df and du

Shell
df -h /var/lib/myapp
Code
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 43M  40M  0     100% /var/lib/myapp

Full. Now find what filled it:

Shell
sudo du -ah /var/lib/myapp | sort -h | tail -3
Code
16K   /var/lib/myapp/lost+found
40M   /var/lib/myapp
40M   /var/lib/myapp/debug-dump.bin

A 40 MB debug dump, written by a forgotten diagnostic flag. If du had found nothing, the next check would be lsof +L1 for deleted-but-open files (Day 14), then df -i for inodes.

A repeatable method: confirm the symptom, check service state, read the logs, check resources such as disk, memory and CPU, find the cause, fix, verify, and record what to prevent.
A repeatable method: confirm the symptom, check service state, read the logs, check resources such as disk, memory and CPU, find the cause, fix, verify, and record what to prevent.

5. Fix and verify

Shell
sudo rm /var/lib/myapp/debug-dump.bin
df -h --output=pcent /var/lib/myapp
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/checkout
Code
  1%
200

Restored. Confirm the fix with the same check that showed the symptom.

Why it didn't fail sooner

Reproducing this incident took three attempts. With the disk 100% full but the app appending short lines to an existing file, requests still succeeded: each small write fit into space already allocated to the file's last block. Requests only failed once the app needed new blocks (here, a new file per session). The lesson: a full disk doesn't fail everything at once; it fails whatever needs new space next, which can be the database, the logs or the next deploy. Alert well before 100% (Day 14).

A general method for troubleshooting Linux servers

  1. Symptom: what exactly fails, for whom, since when? Reproduce it.
  2. Recent changes: deploys, config, traffic, certificates, cron jobs.
  3. Service state: systemctl status, restarts (NRestarts), crash loops.
  4. Logs: the journal and application logs around the start time.
  5. Resources: CPU and load (uptime, top), memory (free, OOM kills in journalctl -k), disk space and inodes (df -h, df -i), I/O (vmstat, iostat).
  6. Network: listening sockets, connectivity to dependencies, DNS (Day 22).
  7. Narrow down: change one thing at a time, and verify each hypothesis.
  8. Fix, verify, then prevent: an alert at 85% disk, the debug flag removed from production config, the app's data on its own filesystem (which is why the root filesystem stayed healthy here).

Write a short postmortem with the timeline and what will prevent a repeat (see the CI/CD course on blameless postmortems).

Check yourself

Active only means the process is running. The app was running but failing every request because it couldn't write to disk, so you need to test the real behavior, like the endpoint itself.

ENOSPC: no space left on the device for the write. It points straight at disk capacity or inode exhaustion, which df -h and df -i confirm.

Small appends fit into space already allocated to the file's last block, so they succeeded. Failures began only when the file needed new blocks, which explains why the problem appeared gradually rather than at the moment the disk filled.

Confirm the symptom, check service states, read the logs, check resources like disk, memory and CPU, then fix and verify, changing one thing at a time.

Find what filled it, add log rotation or retention, and set an alert at a safe threshold so it doesn't happen again.