I am an idiot.

For most of today, I have been trying to find a way to recover almost 60 GB of images that I’m pretty sure have been sacrificed to the MTBF gods. A little while ago I ran into a bit of a problem where my Aperture library was growing too large for my laptop’s primary disk, so I moved it onto an external drive temporarily while I could figure out a backup solution.

I had planned on implementing at least a partial backup solution at some point within the next couple of weeks (something about which I will write a little later in this post). I wanted to take the time to properly plan it, ensure it was redundant and as bullet-proof as I could make it, and then fully implement it and get it ready to go.

Unfortunately, this is exactly the time when that external hard disk decided it didn’t feel like working anymore.

I’d been putting off trying to recover the information for the last couple of weeks as I knew something was wrong with the drive but I wasn’t entirely sure what that was. All the diagnostics I ran on the drive came back OK. But now, it’s seeking and reading data extremely slowly, in short “bursts” every couple of seconds or so. It just took me about 20 minutes to transfer five RAW files while I have been typing this.

The library that’s been hosed actually contains most of the images that I’ve taken with my DSLR over the past year, including my Happiness trips to Lisbon and Vienna (half of the Lisbon pictures I uploaded long ago and are theoretically accessible, but in JPG format), and lots of pictures of the kids. Thankfully, my wife has pictures of her own from our point-and-shoot that are in an iPhoto library that is backed up to a Time Machine, but I’ve lost quite a few of my personal photos that I’ve taken in various places while traveling for work.

I suppose I should be thankful I hadn’t yet merged those libraries.

So I’m sitting here tonight, moving these pictures one by one from the dying drive to a new one, and as I do this I realize that the only person I’m upset at in this whole thing is myself for being dumb enough not to follow my own advice as quickly as possible. I know from experience that you need redundant storage for anything important because things can and do break.

And as I try—probably in vain—to rescue these files, I have been trying to come up with the plan for backing everything up. If anyone reading has any ideas for something I missed, please let me know in the comments as I’m most interested in making sure this is pulled off appropriately to I don’t have to do this again.

And before you ask, I looked into data recovery solutions. It looks pretty likely we’re talking a couple grand to do that here. I love my photos, but I don’t love them that much.

First, the in-home solution:

• NAS appliance here, in the basement, four drives, can’t decide between RAID 0+1 and RAID 6. I’m sure there’s someone with more experience on this one.
• Store everything, or at least copies of everything, important on both the local computer and the SAN.
• Continue to Time Machine the family computer (if the backup drive goes kaput, I should have enough time to replace it, but I’d like to get a rotation going with this at some point). Never mind that the iMac that serves as this machine seems to be having some serious problems with the video hardware.

Second, the off-site solution. I’m not quite sure what to do with this. This is primarily desired for our photos, as we’re approaching probably half a terabyte of them and don’t want to lose them. My first thought was to find an external server I could use for the storage, then create a Subversion repository there, check it out on my local machine, and sync my Aperture library using svn, as it’s really just a big directory. I could automate this as often as I want, probably daily.

I think that’s the best thing I’ve come up with. If you know of a service that would fit this, I’m all ears. I thought at first that S3 would be useful, but to be honest the way AWS stuff works is a bit cryptic to me and I’m not sure that I know what I’m doing enough to make it work properly. I could lease some hosting from someone, but that’s going to come with all kinds of services installed that I don’t actually need.

What I need is a big disk in the sky where I can make a backup of a specific set of folders on a nightly basis, upload only the things that have changed in those things (including the Aperture library bundle), and easily pull it down in a case where I’ve lost everything. It’s possible that doesn’t exist or that I’ll have to finagle my own solution from something not completely ready-made.

All I know is that I need to find a way not to make the same stupid mistake again with these types of files. Because as I type this, the file transfer on the latest five RAW images I’m trying to rescue has died. And there are over 6,000 masters in this Aperture library.