Repairing FreeNAS after a power outage (FreeNAS rebooting)
For reasons beyond me, I have a storage server set up with FreeNAS. I find it convenient to have all of my movies/music/TV shows on a PC that sees no action besides reading and writing to its drives. This weekend, while streaming a show from my storage server, the power went out. The server is connected directly to mains, so it went down – hard. To be fair, my Ubuntu based media center PC also died, never to recover. After a bit of troubleshooting and an annoying secondary problem, it seems to be up and stable.
My FreeNAS server is based on a 4 year old AMD 3200+ PC. Before setting everything up I made sure to clean the motherboard and CPU mount, applied new thermal grease on the CPU heatsink and upgraded the PSU to a Thermaltake 480W PSU – I really wanted to reduce chance of having any electrical/heat related issues that would take out the system. FreeNAS was loaded on a USB drive and a single 1.5TB drive was attached to the SATA bus. The server sat alone in a closet, connected via Gigabit ethernet to my home LAN and remained there. Headless, keyboard-less and monitor-less.
After the outage, the server did not respond to pings. I hooked it up to my monitor and attached a keyboard. It seemed like the BIOS did not even attempt to load an OS. I checked the BIOS settings and it seemed to revert to booting from HDD. Triumphantly I changed it to boot from USB and rebooted. Nothing. I then went ahead and restored BIOS defaults and again changed to boot from USB, just to start from a clean slate seeing as something freaky went on with the BIOS settings. No dice.
I tried yanking the USB key holding FreeNAS but that showed nothing. I then yanked the SATA cable from the HDD and finally something – the BIOS seemed to print something about Nvidia boot loader agent, but again hung. I knew something had shifted so I replugged the USB drive and voila – FreeNAS loaded. I let it load again with the HDD connected and it seemed I was successful.
Plot twist: FreeNAS reboots itself every 20 minutes
I patted myself on the back, called the wife, sat on the couch and loaded “The Wire” episode we were watching. After a few seconds, VLC crapped out indicating failure to read from file. The storage server was down again. I was smart and pessimistic enough not to disconnect the server from the monitor so I could get a glimpse of what was happening. After a bit of investigating, I saw that FreeNAS would reboot every ~15-20 minutes, claiming the following panic:
panic: ffs_blkfree: freeing free block
So it’s a file system issue, methinks. I then performed the following:
- Via the FreeNAS shell I ran “mount” to see which partition my storage mount was on (it was /dev/ad4p1)
- Via the FreeNAS web interface, I deleted the shares on the drive and deleted it from the drive management (this does not destroy the data on the drive, just detaches it apparently). Reboot
- Manually unmount mount in /mnt (in my case, “umount /mnt/storage1″ – this depends on your setup). Reboot. At this point the partition should not be in use by FreeNAS
- Time to fix the partition. I ran “fsck -t ufs -y /dev/ad4p1″ several times, until no errors appeared. Reboot
- Via the FreeNAS web interface, added the drive (making sure pre-formatted was set to UFS). Re-added the share
That’s it. FreeNAS is up, running and stable… until the next power outage that is.