FreeBSD run out of inodes
So you're running
freebsd-update fetch install or
portupgrade -a and suddenly it dies – no space left on device. Running
df tells a different story:
# df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ada0s1a 1031708 419328 529844 44% / devfs 1 1 0 100% /dev /dev/ada0s1b 503580 190140 273156 41% /var /dev/ada0s1d 6162708 4004416 1665276 71% /usr devfs 1 1 0 100% /var/named/dev
We've run out of inodes:
# df -ih Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ada0s1a 1G 409M 517M 44% 3.4k 62k 5% / devfs 1.0k 1.0k 0B 100% 0 0 100% /dev /dev/ada0s1b 491M 185M 266M 41% 592 31k 2% /var /dev/ada0s1d 5.9G 4.0G 1.4G 71% 425k 0 100% /usr devfs 1.0k 1.0k 0B 100% 0 0 100% /var/named/dev
There is still 1.4GB free on
/usr, but no more files can be created. This is because there are thousands of small files in places like
/usr/ports. Each file takes up one inode however large or small the file. By default, FreeBSD creates one inode for every four kilobytes of data (if your disk has 4k sectors). This can be altered, but only when creating a filesystem, it can't be change retrospectively. This would have been useful to know at install time, but it's too late now!
This particular computer has only an 8GB disk because it is an Alix board with its filesystem on a compact flash card, so there is no unallocated space we can use. However, with some jiggery-pokery we can temporarily move the important parts of
/usr to the root partition, make a new filesystem on /usr and put all our stuff back.
First of all back up the system. It's all to easy to make a typo and newfs or rm a filesystem you didn't mean to.
Now, let's get rid of the non-essentials.
/usr/obj take up loads of space and can be recreated later:
# cd /usr && rm -rf src ports obj
/usr should be a bit more manageable now:
# du -s -h /usr 488M /usr
Great, we can just about fit that onto the root partition. Use
cpio to do so:
# mkdir /tempusr # cd /usr # find . -print -depth | clio -dpam /tempusr
Splendid. At this point you will need local access to the machine, it's time to drop to single user mode:
# init 1
With all the daemons stopped, it should be possible to unmount
# umount /usr
Make the new filesystem with more inodes:
# newfs -i 1024 /dev/ada0s1d
Remount and copy the precious data back:
# mount /dev/ada0s1d /usr # cd /tempusr # find . -print -depth | cpio -dpam /usr
Give it a reboot and everything should be ship shape:
# shutdown -r now
Now that the system has rebooted, there will be plenty of inodes available:
df -ih Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ada0s1a 1G 708M 218M 76% 19k 46k 29% / devfs 1.0k 1.0k 0B 100% 0 0 100% /dev /dev/ada0s1b 491M 185M 266M 41% 592 31k 2% /var /dev/ada0s1d 5.6G 1.4G 3.7G 28% 106k 1.5M 7% /usr devfs 1.0k 1.0k 0B 100% 0 0 100% /var/named/dev
/tempusr isn't as straightforward as you might hope. Here's the trick:
# cd / # chflags -R no schg tempusr # rm -rf tempusr
/usr/obj will be regenerated automatically) back just follow the same procedure as updating FreeBSD.