Repairing FreeNAS after a power outage (FreeNAS rebooting)

For reasons beyond me, I have a storage server set up with FreeNAS. I find it convenient to have all of my movies/music/TV shows on a PC that sees no action besides reading and writing to its drives. This weekend, while streaming a show from my storage server, the power went out. The server is connected directly to mains, so it went down – hard. To be fair, my Ubuntu based media center PC also died, never to recover. After a bit of troubleshooting and an annoying secondary problem, it seems to be up and stable.

My FreeNAS server is based on a 4 year old AMD 3200+ PC. Before setting everything up I made sure to clean the motherboard and CPU mount, applied new thermal grease on the CPU heatsink and upgraded the PSU to a Thermaltake 480W PSU – I really wanted to reduce chance of having any electrical/heat related issues that would take out the system. FreeNAS was loaded on a USB drive and a single 1.5TB drive was attached to the SATA bus. The server sat alone in a closet, connected via Gigabit ethernet to my home LAN and remained there. Headless, keyboard-less and monitor-less.

After the outage, the server did not respond to pings. I hooked it up to my monitor and attached a keyboard. It seemed like the BIOS did not even attempt to load an OS. I checked the BIOS settings and it seemed to revert to booting from HDD. Triumphantly I changed it to boot from USB and rebooted. Nothing. I then went ahead and restored BIOS defaults and again changed to boot from USB, just to start from a clean slate seeing as something freaky went on with the BIOS settings. No dice.

I tried yanking the USB key holding FreeNAS but that showed nothing. I then yanked the SATA cable from the HDD and finally something – the BIOS seemed to print something about Nvidia boot loader agent, but again hung. I knew something had shifted so I replugged the USB drive and voila – FreeNAS loaded. I let it load again with the HDD connected and it seemed I was successful.

Plot twist: FreeNAS reboots itself every 20 minutes



I patted myself on the back, called the wife, sat on the couch and loaded “The Wire” episode we were watching. After a few seconds, VLC crapped out indicating failure to read from file. The storage server was down again. I was smart and pessimistic enough not to disconnect the server from the monitor so I could get a glimpse of what was happening. After a bit of investigating, I saw that FreeNAS would reboot every ~15-20 minutes, claiming the following panic:

panic: ffs_blkfree: freeing free block

So it’s a file system issue, methinks. I then performed the following:

  1. Via the FreeNAS shell I ran “mount” to see which partition my storage mount was on (it was /dev/ad4p1)
  2. Via the FreeNAS web interface, I deleted the shares on the drive and deleted it from the drive management (this does not destroy the data on the drive, just detaches it apparently). Reboot
  3. Manually unmount mount in /mnt (in my case, “umount /mnt/storage1″ – this depends on your setup). Reboot. At this point the partition should not be in use by FreeNAS
  4. Time to fix the partition. I ran “fsck -t ufs -y /dev/ad4p1″ several times, until no errors appeared. Reboot
  5. Via the FreeNAS web interface, added the drive (making sure pre-formatted was set to UFS). Re-added the share

That’s it. FreeNAS is up, running and stable… until the next power outage that is.

9 comments


  • THANK YOU! This worked to recover my data after my machine got corrupted during a lightning storm and power outage. Thanks so much for posting this!!!!

    July 26, 2011
  • Caspar

    Many thanks. Before getting the panic: ffs_blkfree error I got another error directly after starting up Freenas, freezing up the whole system. To solve this I’ve used the Live CD to reinstall Freenas. In the end your guide to solving the above panic brought my Freenas box back to live. Thanks again!

    August 4, 2011
    • Eran Duchan

      Great to hear. 6 months later and freenas is still up and running, no file system issues to report.

      August 4, 2011
  • Magnus

    Yes, thank you very much for posting this. I googled “freenas reboot” and got this solution right away and since i also had several power outages i gave the solution a shot and it worked instantly! Perfect! My mind was set on replacing the PSU, the motherboard, the USB stick i’m booting from etc so this was a lot cheaper! :)

    September 13, 2011
  • Martin Trepanier

    Thanks dude!

    You saved my day! Was about to try to backup thing on an external harddrive to reformat, but was crashing every 10 to 15 minutes.

    Now everything is fine

    I had to unmount my storage thru the webgui, because couldn;t do it in shell mode, don’t know why.

    But after that, everything went fine. Had to run fsck 3 or 4 times to make everything clean!

    February 5, 2012
  • Jonathan Vella

    I have been using freenas 8.04 quite a while and had 2 mirrored 500Gb Hd installed. All of a sudden the network slowed down and was barely usable so I restarted the nas but to my surprise it would not link up the lan. Any ideas please as I’m desperate for the date! I would appreciate and offer payment for it no worries.

    May 22, 2012
  • Ed

    I didn’t get the panic but wasn’t able to access the share anymore. Lots of time outs etc. Executed the procedure and all works like a charm now. What is unclear is whether all the repairs corrupted files or not. So far no indication but I haven’t checked all of the 2TB (will take a while ;-)
    Thanks for sharing this procedure.

    June 11, 2012
  • Hi Eran, i got problem for 2 days ago till now,
    the problem is begin from when i start to change
    folder sycon client (Using Delta Copy), and suddenly my freenas web cannot login, then i reset the webgui pasword, but the freenas become hang, and i rebort the frenas and problem come that freenas say cannot mounted properly to ufs: dev/ad01a ( to 1 TB external Hardisk) is there any way to solf this , actualy i have already backup the configure setting frenas, but My Freenas cannot boot couse cannot mounted to my external hardisk

    July 27, 2012
    • Eran Duchan

      Sorry, Dudun… I have no idea. Looks like a case for google (or a reinstall). Worst comes to worst you can always mount UFS under Linux to get your data back.

      July 27, 2012

Leave a comment


Name*

Email(will not be published)*

Website

Your comment*

Submit Comment

© Copyright pavius.net: no refunds - Designed by Pexeto