Incident: Full Disk Space

Description

A server's disk (often /, /var, or /home) is full, causing system instability or service crashes.

Symptoms

• df -h shows 100% usage
• Writes fail with No space left on device
• System logins may fail

Root Cause Checklist

• Unrotated logs or core dumps?
• Growing Docker volumes or images?
• Backup or temp files not cleared?

Resolution Steps

1. Check usage:

   df -hT
   du -sh /* 2>/dev/null
   

2. Identify large files:

   ncdu /
   

3. Clean logs and caches:

   journalctl --vacuum-time=7d
   rm -rf /var/cache/*
   

Preventive Actions

• Set up logrotate
• Monitor with Prometheus/Grafana
• Clean Docker system regularly

Tools & Commands

• df, du, ncdu, journalctl, docker system prune