Craig Mayhew’s Blog

Regular backups, we all know the score. If you don’t make regular backups then you will lose data.

In recent years more and more of the big players are releasing bigger and cheaper cloud solutions. e.g. Amazon’s EC2 or Microsoft’s Azure. Clouds offer incredible scalability allowing you to go from needing no processing power to thousands of CPUs or a few megabytes of storage to terabytes without ever needing to upgrade your server. Unfortunately sometimes proper backup solutions are not put in place. After all it is no easy task to backup an entire data centre in a timely manor that doesn’t impact upon the clouds performance. Even more than that the backup must be stored differently to the main cloud. For instance it’s not a good idea to backup an Amazon EC2 cloud to another EC2 cloud. If someone found an exploitable flaw in the EC2 platform that meant they could wipe the cloud or the cloud itself failed and erased all it’s data then the exact same thing could happen to the backup copy. A simple solution is to mirror your data between two cloud platforms. The perfect solution is to maintain a tape backup (or at least some kind of offline storage) of your data and keep it in geographically different place.

Cloud computers are not immune to failure, they are simply as good as the software running on them. Some good examples of massive irreversible data loss:

Upto 1 million T-Mobile sidekick customers lose their mobile data

100,000 websites wiped by hack attack

From this day forth I see the Drobo as useful as volatile storage when it comes to long term storage of my data.

How it happened:
I unplugged the Drobo from an old computer using the now ancient USB 1.1 and plugged it into my Ubuntu computer. The Ubuntu computer gave a message saying it couldn’t mount the volume. So I plugged it into my laptop to see what Vista had to say, but unfortunately Vista refused to mount the drive as well. After checking the drive in Vista’s disk manager I came to conclusion that Vista saw it as a RAW partition! It had completely forgotten that it was an NTFS partition so Vista didn’t know what to do with it.

My immediate thought was the Drobo is going to start freeing up space and wiping the drives sector by sector. Fortunately I couldn’t hear any disk activity so I thought to myself that I may be able to recover the data. I used a copy of EASUES Data Recovery to do a scan of the hard disk to find lost partitions. It took 48 hours to do the scan, which is ridiculous when you consider I only had 300GB of data on the Drobo and the Drobo’s entire capacity was only 500GB, 48 hours is fairly ridiculous at 1.7Mb/s.

At this point my laptop where I was recovering the data to, crashed. Not a problem as I had my initial scan saved so I could continue where I left off. I wandered off for 2 minutes to come back and find my computer had started a file system scan of the Drobo and it had attempted to repair damaged sectors!! This could be really good or really bad. After windows booted and I franticly continued the scan I ran into another problem! The scan had not remembered where it had stopped so I had to leave it running for a further 48 hours! This time around however I had the mortifying experience of watching the little blue lights go out one at a time on the Drobo’s front panel. It appears that the file system “fix” that vista had run had in fact instructed the Drobo to truncate the “problem” parts of the disk. This consequently meant that I was running the scan for the second time and that this time the Drobo was probably slowly destroying my data. As it turned out the file system scan had destroyed a lot of data but in the end I managed to get a portion of my data back.

Important lessons:

• If you delete something from a drobo, your data is gone… forever.
• If you Drobo file system gets damaged, do not let windows or any other operating system repair it or you will lose some or all of your data.
• The drobo is slow, you can’t recover data in a timely manner (if you can recover it at all).
• Don’t trust the Drobo, as it is only a single device it cannot provide an ultimate solution (or even a reliable one). I will be moving to some kind of distributed private cloud storage setup and will blog about that when I’ve got one going.
• During the time that I have owned the Drobo, it has failed twice. No other disks, flash drives or dvds have died in my house since I bought the drobo. So the Drobo fails more often than any of my other storage devices.