Monday, March 19, 2007

Backing up is hard to do


I have a problem. I take pictures. Lots of them. Gigabytes of pictures. Many of them I actually quite like and I'd like to keep them around. But I shoot digitally and the one constant fact for digital storage is that it fails. All the time. Hard drives are just little time bombs spinning away, ready to destroy your data. So, at the first level, I make two copies of the images when I load them on to my computer. Straight from the memory cards they get copied once to a directory on my local PC and once into a Network Attached Storage (NAS) device. The local hard drive on the PC is a fast RAID 5 set-up, to give speed and some redundancy. Those are the copies I work on. The larger (1 Tb) NAS box is at the end of a 1Gb ethernet connection, which still somehow ends up too slow to work on. But it is fine for secondary online storage. When I work on an image, I store the final Photoshop version, with all the associated layers. This lets me go back and edit an image or easily recreate a print at different sizes. I've been caught out before by not keeping these files, thinking I could just recreate an image from the original RAW file. Then someone wanted 40 images with a short deadline and I had to turn down the opportunity - so now I keep the PSD files. Those can be 250Mb each. They get sync'ed to the NAS box as well, using Unison. I also occasionally go through and cull the terrible images. I don't believe in keeping everything from every shoot - there is plenty of bad stuff to cut out. But that takes time and if you are in a hurry, you have the opportunity to delete an important image. So the lazy approach is to not clean out things often enough.
So with all that, I must feel pretty secure, right ? Well - not really. Now I have a couple of copies of my images, so that's good. Except the drive space on the local PC is a bit tight, so I occasionally have to delete images from there. So I'm left with one copy on the NAS box. Now it is RAID too, so I have some protection if a drive fails in there, but I still only have one copy in the online storage. A virus on my machine could go on a rampage across the network corrupting files. Something less malicious but more common is that I could do something stupid and delete the files. Lighting could strike the power cable (Texas has a whole lot of thunderstorms) and fry my computer and the NAS box. Sure I have surge protection and UPS on all the power supplies. But not the network cables, so there is still a vector. I don't think I really trust the $30 surge protection from Fry's anyway.
So I make backups onto DVD. Well, occasionally I do. Problem is I can easily produce enough data to fill 2 DVDs each time I take my camera out. Sitting there, backing up files to DVD is a fairly long & boring chore so it doesn't get done as often as it should. Then those DVDs can suffer from the usual bitrot of all digital media. So to be on the safe side, you really need to burn 2 copies of each DVD. Suddenly that is 4 DVDs any time I pick up the camera and several hours of popping back to the PC to swap over disks. Then the pile of disks starts to well, pile up.
Even at that point, now that I've got my RAID backup & NAS and stack of (mostly fictional) DVDs. I'm actually thinking of buying another NAS box and only occasionally powering it up and connecting it - doing a weekly or monthly backup of the online NAS box to that as a stop-gap solution. But then I worry about those hard drive platters that are sitting non-spinning and worry if they'll start up each time or what that sort of operation does to the Mean Time Before Failure (MTBF) of the hard drives.
What happens if the house catches fire ? Or we have another typical Texan biblical flood ? Or if we get burgled ? So maybe those fictional DVDs need to be stored out of the house - but I don't know where that would be. We have a small safety deposit box in the bank, but it isn't going to fit all those DVDs. A larger disk format would be fantastic. Blu-ray or HD-DVD promise 25Gb or 50Gb writable disks. That's just about perfect for my needs right now. I just wish it was a bit more affordable and more importantly standardised. But then - do I need to do duplicate copies of the blu-ray DVDs ? Is there any faith that they won't suffer the same fate as DVDs or CDs now ? A few years lifetime in careful storage before the errors start to appear ?
I suppose I could shoot less, but why would I want my photographic process to be dictated by the availability of secure storage ? I've yet to see a solution that would give any confidence that the images were secure from accidental deletion when online or store enough for easy off-line maintenance. Please, please, please - if you have a good solution - let me know!


Kate said...

Hi Gordon,
No great solution, but an additional suggestion. I've been offloading my images onto an 500M external harddrive. Powering it on and adding the monthly new stuff only. When it gets filled, I start another. This can be stored offsite pretty easily, and isn't susceptible to power fluxuations. Firewire/USB2 connections mean that it certainly isn't usable for editing, but as a long term storage and adjunct to the systems you have in place, it might be worth considering.

I've got DVDs burned as well... and organizing the stacks is starting to become a concern. - Kate

gabriel said...

The only perfect solution is to make archival prints my man!
It was a pleasure meeting you last weekend and shooting as well! Keep the creative hearts and juices flowing!

advman said...


This may not help you very much, but I solve the problem by keeping everything on hard drive, on two computers in two places. During the week I live in Vienna, on weekends I am in Carinthia, Austria's southern province. Before leaving either location, I always search for the new files of the week or weekend, put them uncompressed into an archive file on my 60GB music player, keeping paths intact, and the first thing upon arrival is to restore the archive to the other computer.

DVDs simply are a desaster. I had the plan to make DVD backups, but I've given it up. Blu-Ray or HD-DVD may seem big now, but who knows how much space we'll need by the time they've become affordable? And not only that. I can imagine them to be faster than DVDs when burning, but 5 times faster? Never!

Thus, on each individual computer I'm back to hard drive storage again. Recently, instead of buying a Blu-Ray burner for 600+ EUR I've simply bought two 500 GB S-ATA drives for 135 EUR each, one for each location. This will suffice for a year, and then I intend to buy a pair of 1 TB drives.

Basically I live with the risk of losing 2 or 5 days work, and I can live with that. Key to the whole scheme are simply the two locations and the weekly routine. No RAIDs, no other redundancy, only 300 km in between.

Ted Byrne said...

Yeah, you can get pretty weeeerd thinking about crashing drives. Or getting hit by a comet... or... or... But it can happen. So I keep a 500g drive next to my desk in my office at work, and another in my studio at home. I have an external 150g portable HD that's the size of pack of playing cards that's firewire compatible with both of my macs and chains with the drives at both locations. Every couple of days I drop stuff from one or the other of the drives and take the files in each direction. I use the same method to backup my MacBookPro to each drive.

I figure that the possibility of both collapsing simultaneously is way beyond a comet hitting each of them at the same time. Seems sufficient. Once I started a CD then a DVD backup system but it just took took too long. Plus it's too damned expensive. The point of digital photography is that once the capital goods are in place, the film's free. But if you have to think in terms of purchased DVDs... Film starts to sound plausable again... well almost.

Oh yea... on the road I backup my cards in the evening to the PowerBook. When the image is particularly valuable I can also back it up to the portable drive. With a combined storage of about 225g in the PowerBook plus the portable drive... Well I'm covered through most shoots. Heck... I'm covered through all shoots...

A last point... When I'm doing work for my magazines from a remote location, I've been known to file the best back to the office in the evening as jpg files. And I'll send a cc to my home account as well. That's a lot of confidence.

And it's all pretty cheap.

Thanks for sharing

My Images Explained
My Images Stored