Troubleshooting a Production Linux Server
On this page
The incident: checkout returns 503
Checkout pages return errors. On the server, systemctl status says both nginx and the app are active (running). Nothing has been deployed today. This walkthrough was reproduced on the course's Ubuntu VM: nginx in front of the app from Day 19, with the app's data on its own small filesystem.
1. Confirm and characterize the symptom
for i in 1 2 3; do curl -s -o /dev/null -w '%{http_code} ' http://127.0.0.1/checkout; done503 503 503Consistent, not intermittent. Every request fails.
2. Check the services with systemctl
systemctl is-active myapp nginxactive
active"Active" only means the process is running, not that it works. Health checks that make real requests (Day 26) catch what status checks miss.
3. Read the logs with journalctl
journalctl -u myapp --since "-10min" -o cat | tail -5ERROR writing session: [Errno 28] No space left on deviceErrno 28 is ENOSPC. The app fails to write session data and returns 500, which nginx turns into 503.
4. Check the disk with df and du
df -h /var/lib/myappFilesystem Size Used Avail Use% Mounted on
/dev/loop0 43M 40M 0 100% /var/lib/myappFull. Now find what filled it:
sudo du -ah /var/lib/myapp | sort -h | tail -316K /var/lib/myapp/lost+found
40M /var/lib/myapp
40M /var/lib/myapp/debug-dump.binA 40 MB debug dump, written by a forgotten diagnostic flag. If du had found nothing, the next check would be lsof +L1 for deleted-but-open files (Day 14), then df -i for inodes.
5. Fix and verify
sudo rm /var/lib/myapp/debug-dump.bin
df -h --output=pcent /var/lib/myapp
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/checkout 1%
200Restored. Confirm the fix with the same check that showed the symptom.
Why it didn't fail sooner
Reproducing this incident took three attempts. With the disk 100% full but the app appending short lines to an existing file, requests still succeeded: each small write fit into space already allocated to the file's last block. Requests only failed once the app needed new blocks (here, a new file per session). The lesson: a full disk doesn't fail everything at once; it fails whatever needs new space next, which can be the database, the logs or the next deploy. Alert well before 100% (Day 14).
A general method for troubleshooting Linux servers
- Symptom: what exactly fails, for whom, since when? Reproduce it.
- Recent changes: deploys, config, traffic, certificates, cron jobs.
- Service state:
systemctl status, restarts (NRestarts), crash loops. - Logs: the journal and application logs around the start time.
- Resources: CPU and load (
uptime,top), memory (free, OOM kills injournalctl -k), disk space and inodes (df -h,df -i), I/O (vmstat,iostat). - Network: listening sockets, connectivity to dependencies, DNS (Day 22).
- Narrow down: change one thing at a time, and verify each hypothesis.
- Fix, verify, then prevent: an alert at 85% disk, the debug flag removed from production config, the app's data on its own filesystem (which is why the root filesystem stayed healthy here).
Write a short postmortem with the timeline and what will prevent a repeat (see the CI/CD course on blameless postmortems).
Check yourself
Active only means the process is running. The app was running but failing every request because it couldn't write to disk, so you need to test the real behavior, like the endpoint itself.
ENOSPC: no space left on the device for the write. It points straight at disk capacity or inode exhaustion, which df -h and df -i confirm.
Small appends fit into space already allocated to the file's last block, so they succeeded. Failures began only when the file needed new blocks, which explains why the problem appeared gradually rather than at the moment the disk filled.
Confirm the symptom, check service states, read the logs, check resources like disk, memory and CPU, then fix and verify, changing one thing at a time.
Find what filled it, add log rotation or retention, and set an alert at a safe threshold so it doesn't happen again.