Disk Full: How to Find and Effectively Delete Large Directories Using Linux

There are 4 steps to effectively free disk space when a partition is full on a linux machine:

1. Find the parition that is full (with df -h)
2. Find the largest directories (with du -kx /mountpoint | sort -n | less)
3. Delete files
4. Make sure no more file handles exist (with ls -l /proc/*/fd | less)

Find the partition that is full

The df command shows you how much space there is available on each parition. Using the -h option formats the output in a way that's easier to read, which looks as follows:

$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/sda5                    147G   99G   41G  72% /
udev                          12G  4.0K   12G   1% /dev
tmpfs                        4.7G  976K  4.7G   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                          12G  156K   12G   1% /run/shm
/home/tsteffens/.Private     147G   99G   41G  72% /home/...

The 'Use%'-column you will show you which partition is too full and the 'Mounted on'-column will show you where it is mounted.

 

Find the largest directories

The command du allows you to print out all directories together with the sum of the sizes of all files in each directory (including subdirectories). To determine the largest directories for a single partition, you can use:

du -m -x /mountpoint | sort -n

This will produce an output as follows:

$ du -m -x /home/tsteffens/VirtualBox\ VMs/ | sort -n
...
1670    /home/tsteffens/VirtualBox VMs/vagrant_sar_1386255963
13175   /home/tsteffens/VirtualBox VMs/vagrant_db_1385708315
15936   /home/tsteffens/VirtualBox VMs/vagrant_db_1386167954
30780   /home/tsteffens/VirtualBox VMs/

The -m parameter tells du to print out the size in megabytes. -x skips directories on a different partition. sort -n sorts the lines by the first number in each line.

You might want to pipe the result to less for a better overview:

du -mx /mountpoint | sort -n -r | less

With the -r option sort reverses the order and starts with the highest value first.

Delete files

Once you have selected the directories you want to delete, you should have a look at the content using ls (-l tells ls to output the long format containing additional information like size and owner):

$ ls -l /whatever-directory
total 16317360
drwx------ 2 tsteffens tsteffens        4096 Dec  5 11:02 Logs
-rw------- 1 tsteffens tsteffens 16708927488 Dec  5 18:29 box-di...
-rw------- 1 tsteffens tsteffens        7617 Dec  5 18:29 vagran...
-rw------- 1 tsteffens tsteffens        7617 Dec  5 11:02 vagran...

This will show you which files are the biggest ones - in case you do not want to delete the whole directory. But there's another information that is easily overlooked: the number in the second row. It tells you how many hard links exist to that file.

If there should be more than one hard link to that file (i.e. the number in the second row is greater than 1), you need to make sure all other hard links to that file are deleted as well. Although this is a rather rare case, you can encounter it especially when working with version control or backup systems.

You can find all hard links to a file as follows:

find /mountpoint -xdev -samefile someFile

The -xdev option tells find NOT to traverse directories on other partitions. If you really want to delete all of them you can use:

find /mountpoint -xdev -samefile someFile | xargs rm

For directories it is absolutely fine to have more than one hard link to themselves: the . link in the directory itself (which you can see when appending -a to a ls call) will always increase the hard link count by one and each .. link in a subdirectory will do so as well.

If you shouldn't find any additional hard links, just go ahead with deletion using rm for files or rm -rf for directories.

Make sure no more file handles exist

After you have deleted some files, you may call df -h again to check how much new space you have won. Don't panic if this is much less than you expected. Unlike under Windows (where a file accessed by a program can not be deleted), Linux does not prevent deletion of files currently in use! But as long as the file is in use, the space for that file will not be available again.

To find out about that, you can make use of the proc file system as follows:

ls -l /proc/*/fd | less

This will list all open file handles of all processes. Now you can search for the file you have deleted (using cntrl-/ [downward seach] or cntrl-? [upward search]). If a process still uses it, you will find something like:

/proc/5150/fd:
total 0
lrwx------ 1 tsteffens tsteffens 64 Dec 15 18:08 0 -> /dev/...
lrwx------ 1 tsteffens tsteffens 64 Dec 15 18:08 1 -> /dev/...
lrwx------ 1 tsteffens tsteffens 64 Dec 15 18:08 2 -> /dev/...
lr-x------ 1 tsteffens tsteffens 64 Dec 15 18:08 3 -> /dev/tty
lr-x------ 1 tsteffens tsteffens 64 Dec 15 18:08 4 -> /home/tsteffens/tmp/test (deleted)

Now you know which process still holds a handle on that file - the process id can be found after /proc/ in the first line.

To get more information you can either look at the other proc fs entries in the processes directory (like /proc/5150/cmdline) or use the ps command:

$ ps aux | grep 5150
1000      5150  0.0  0.0  19436   996 pts/2    S+   18:08   0:00 less test
1000      5527  0.0  0.0  15048   620 pts/0    S+   18:47   0:00 grep ...

That tells you that in this case a less command still uses the file.

You can free the disk space by ending the appropriate process. For this you have (at least) 3 options:

1. Stop the process "the way it should be done" (e.g. go to the terminal where the less command is still open an press q, or restart the service with service ... restart etc.)
2. If that should be possible by any reason (e.g. it is no service and it's not opened in a terminal), you can use kill processId (inserting the id of the process for processId). This will give the process the signal that it should end itself.
3. If the above point didn't succeed, you can still use the rough way using kill -9 processId - which terminates the process immediately. Note that the process will not be able to do any cleanup before shutting down.

You can check if the process was ended properly using

ps aux | grep processId

Final Remarks

Please let me know if you should have any questions or suggestions! Many thanks to Andreas Krüger who gave a talk on this topic, which gave me a lot of input for this post!