How to identify large directories for ‘No space left on device’ on Linux

TL;DR

cd / and run

sudo du -sh * --exclude proc --exclude sys --exclude dev

and then repeat for the largest directory shown (by cding to that directory and running the command above)

Long answer

If you get No space left on device errors on Linux, this means that one of your mounted disks has (virtually) no space left to write on.

First, check which device is the one that has no space left:

$ sudo df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.4G     0  2.4G   0% /dev
tmpfs           494M   51M  444M  11% /run
/dev/xvda1       46G   17G   11M 100% /
tmpfs           2.5G     0  2.5G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.5G     0  2.5G   0% /sys/fs/cgroup
tmpfs           494M     0  494M   0% /run/user/1003

Check the Use % column – you can see that the device mounted on / (your root filesystem, i.e. the one where your system is mounted) is full. In 95% of all cases it’s the root filesystem that’s full.

This post only covers the case where your root filesystem is full. In most other cases, it’s either /dev/shm (in which case rebooting your system typically works) or an external drive (in which case you have to figure out for yourself which directories can be deleted).

Probabilistic method:

This quick check tries to identify the most likely candidates first (based on admin experience). Run this in your shell:

sudo du -sh /var/lib/docker /var/lib/mysql /var/lib/postgresql /home/

This might take some time to complete.

Check if one of those directories is so large that it eats up a significant fraction of your drive space. If not, I recommend going forward with the Simple method:

Simple method:

Copy & paste this into your shell:

function findLargestSubdir { cd "$1" && sudo du -sb * --exclude proc --exclude sys --exclude dev | sort -n ; }

then run

findLargestSubdir /

This will tell you which directory is the largest. It might take a long time to compute the size of all directories. This command will only print anything once it’s finished! The numbers are in bytes.

The last element in the list is the largest one!

Now, run the same command inside the largest directory to find the largest sub-directory. For example, if /var happens to be the largest directory, run

findLargestSubdir /var

which will show you the largest directory in /var. Continue checking the largest subdir(s) using findLargestSubdir until you found out what ate up all the space on your disk.

Advanced method: Much quicker but more complex

If it takes more than 2 minutes to compute the size of all directories, I recommend following this interactive procedure as root (sudo):

1. ls -1 to show all files and directories in /. Example output:

   # ls -1 /
   bin
   boot
   dev
   etc
   home
   initrd.img
   initrd.img.old
   lib
   lib64
   lost+found
   media
   mnt
   opt
   proc
   root
   run
   sbin
   snap
   srv
   sys
   tmp
   usr
   var
   vmlinuz
   vmlinuz.old

2. Now run du -sh *. This will try to compute the size of each of those files and directories. Typically this command will stall for a long time when trying to compute the size of one directory (if that directory has a huge number of files in it). For example, when you see the output

   # du -sh *
   17M     bin
   157M    boot
   0       dev
   7.7M    etc

   and then nothing happens for more than 30 seconds, look up the next entry after the last entry in the list ( etc in this example) in the output of ls -1 above. In this example, this would be /home. Since du -sh took so long computing the size of /home it’s very likely (though not guaranteed) that /home is the directory that takes up so much space. Also check the number

3. In whatever directory you found to be a candidate for being the largest directory, run ls -1 <directory> again and check

Note that this method sometimes has a tendency to identify directories that recursively contain many files as opposed to directories whose total size is large. Therefore, you might need to go back in case you can’t identify any directories that eat up a large fraction of your hard drive space.