FreeBSD run out of inodes

From Wiki
Jump to: navigation, search

So you're running freebsd-update fetch install or portupgrade -a and suddenly it dies – no space left on device. Running df tells a different story:

# df
Filesystem   1K-blocks    Used   Avail Capacity  Mounted on
/dev/ada0s1a   1031708  419328  529844    44%    /
devfs                1       1       0   100%    /dev
/dev/ada0s1b    503580  190140  273156    41%    /var
/dev/ada0s1d   6162708 4004416 1665276    71%    /usr
devfs                1       1       0   100%    /var/named/dev

Buh?

We've run out of inodes:

# df -ih
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ada0s1a      1G    409M    517M    44%    3.4k   62k    5%   /
devfs           1.0k    1.0k      0B   100%       0     0  100%   /dev
/dev/ada0s1b    491M    185M    266M    41%     592   31k    2%   /var
/dev/ada0s1d    5.9G    4.0G    1.4G    71%    425k     0  100%   /usr
devfs           1.0k    1.0k      0B   100%       0     0  100%   /var/named/dev

There is still 1.4GB free on /usr, but no more files can be created. This is because there are thousands of small files in places like /usr/src and /usr/ports. Each file takes up one inode however large or small the file. By default, FreeBSD creates one inode for every four kilobytes of data (if your disk has 4k sectors). This can be altered, but only when creating a filesystem, it can't be change retrospectively. This would have been useful to know at install time, but it's too late now!

This particular computer has only an 8GB disk because it is an Alix board with its filesystem on a compact flash card, so there is no unallocated space we can use. However, with some jiggery-pokery we can temporarily move the important parts of /usr to the root partition, make a new filesystem on /usr and put all our stuff back.

First of all back up the system. It's all to easy to make a typo and newfs or rm a filesystem you didn't mean to.

Now, let's get rid of the non-essentials. /usr/src, /usr/ports and /usr/obj take up loads of space and can be recreated later:

# cd /usr && rm -rf src ports obj

/usr should be a bit more manageable now:

# du -s -h /usr
488M    /usr

Great, we can just about fit that onto the root partition. Use cpio to do so:

# mkdir /tempusr
# cd /usr
# find . -print -depth | clio -dpam /tempusr

Splendid. At this point you will need local access to the machine, it's time to drop to single user mode:

# init 1

With all the daemons stopped, it should be possible to unmount /usr:

# umount /usr

Make the new filesystem with more inodes:

# newfs -i 1024 /dev/ada0s1d

Remount and copy the precious data back:

# mount /dev/ada0s1d /usr
# cd /tempusr
# find . -print -depth | cpio -dpam /usr

Give it a reboot and everything should be ship shape:

# shutdown -r now

Now that the system has rebooted, there will be plenty of inodes available:

df -ih
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ada0s1a      1G    708M    218M    76%     19k   46k   29%   /
devfs           1.0k    1.0k      0B   100%       0     0  100%   /dev
/dev/ada0s1b    491M    185M    266M    41%     592   31k    2%   /var
/dev/ada0s1d    5.6G    1.4G    3.7G    28%    106k  1.5M    7%   /usr
devfs           1.0k    1.0k      0B   100%       0     0  100%   /var/named/dev

Hooray!

Removing /tempusr isn't as straightforward as you might hope. Here's the trick:

# cd /
# chflags -R no schg tempusr
# rm -rf tempusr

To get /usr/src and /usr/ports (/usr/obj will be regenerated automatically) back just follow the same procedure as updating FreeBSD.