My Backup Workflow for Large File Repositories

My Backup Workflow for Large File Repositories

We shift files back and forth trying to find the best backup strategy.

But the activity of moving files around only increases our chance of loosing important data.

At one point or another, we’ve all lost a file forever. Maybe it wasn’t a significant tragedy, or maybe it was 10 years of your child’s photos. Or the project due tomorrow. Or your client’s wedding photos. As we generate more and more data in our lives, it becomes increasingly more important to solidify our backup strategy — and stick to it.

Workflow is personal.

Sometimes when folks ask me about workflow, I preface by saying “this might not work for you”. We need workflows that we are comfortable with and simi-automatic — otherwise we won’t do it. Or we forget a step and then make matters worse when trying to play catch-up. But I’m happy to share with you my workflow, let me know in comments if it works for you, or if you have a better system.

3 copies, or it doesn’t exist.

Some of us may be familiar with this concept. Basically it means, if you don’t have 3 copies of a file, in 3 physically different locations, then it doesn’t actually exist. The intent here is to focus your attention on backups before you begin other aspects of your workflow.

I’m always a little tickled when people post images from our group photography events, even before I drive home. But I also wonder if folks have finished the critical backup stage of their workflow before they start editing images.

My workflow, and my recoverable backups

This workflow may not work for everyone. It’s a methodology that works for me and may give you some hints for your own process. For this example, I’m using a workflow from a photography shooting event.

STEP ONE copy files – do NOT import directly from your memory card!

The first thing I do after plugging in my card is “COPY” the images from my card to a staging directory on my computer, usually on my desktop. Many times I short-cut this step and just move the entire directory.

The most important part of this step is making sure ALL FILES are copied over. If you are using slower cards and an inexpensive card reader, the computer can sometimes “lose” connection as files are copied over. The copy process will halt and you might think the process is complete (and format your card not realizing you’re losing LOTS of images).

This is significantly worse if you try to “import” into your image management software (iPhoto, Lightroom, etc). If your card suddenly ejects and instantly reconnects during import, the software will pop up a pretty dialog box saying all files were imported, when actually they were not.

STEP TWO import files into Lightroom.

I wrote a blog post about safe Lightroom Catalog Backups a few months ago, so check out that article for more details. Each time I import a new batch, I create a new directory on my Drobo, a multi-disk storage device. This is where my “working files” will live forever.

So in a file directory called “2013”, I will create a new directory called “2013-10-05 Grand Prix of Houston”. This way when I order by directory name, all of the directories fall into “date order”.

Note that I say “working files” and not “backup files”. Although the Drobo Mass Storage Device system is often used as a “backup” system, I get the best use out of it as my primary working space.

The Drobo is a black box that sits on your desktop. You plug in 4 or 5 empty (and cheap) hard drives, and your computer “sees” it as one drive. As time goes on and you start running out of space, you pop out whichever drive is the smallest, and pop in a new hard drive of any make, model or size. Presto, more hard drive space w/out having to copy files all over the place.

This also makes upgrading to a new computer MUCH easier, since my work lives on my Drobo rather than my computer’s hard drive. So if my computer crashes (or needs to be repaired/replaced), I can plug the drobo into a new computer and I’m off-and-running like nothing happened.

The Drobo is also redundant. If one of the drobo’s drives goes bad (and you know they do/will), the other drives contain enough data that anything lost can be instantly recovered. All I have to do is run down to Best Buy, find a cheap replacement drive and plug it in. Presto. (Ok, not presto, it takes a while for the device to be ready to use again — but at least I didn’t lose any projects or data).

Finally, if the Drobo itself goes bad (mine is going on about 5 years old, so it won’t be long), I can order another from the manufacturer and pop in the old disks. This is very similar to RAID devices. If your RAID enclosure burns out, your data is still safe on the drives. Unfortunately because each manufacturer uses their own modified version of the linux operating system, you are usually forced to buy another RAID enclosure from the same manufacturer, otherwise the drives have to be reformatted. Similarly, if your Drobo device fails, you have to buy another Drobo to continue using the data on those drives.

One of the criticisms for using external storage (like USB drives, etc) is that it slows your workflow. Drobo products you can buy today have Thunderbolt and USB 3.0. I don’t experience a significant issue using the drobo for editing images. However, as mentioned in the other article, I do keep my Lightroom catalog data on my local hard drive.

Ok, enough about Drobo.

STEP THREE

At this point, I have a single copy of the project files, stored on a redundant drive system. This is good and safe, but not quite safe enough. This is where Time Machine comes in. If you are using Windows, there are several Time Machine-esk utilities on the market. Time Machine makes a backup of my computer a couple of times a day. In my configuration settings, I added my Drobo to the list of things being backed up.

This means I have one huge Time Machine, and eventually I’ll have to revisit this step. But for now, its a completely hands-off backup system. I now have two copies of my project: a copy on redundant file protection system, and a backup on Time Machine.

STEP FOUR

It’s great that I have two copies. Unfortunately they are sitting right next to each other. If my house burns down, or hurricane or tornado, etc, then both will be lost. The third location needs to be “off site”, somewhere completely different. Some folks use two Time Machines, and leave one at the office. Then they just swap them out once a week or month.

I’ve also heard of people using a safety deposit box. But that means having to retrieve it every time you want to update the backup. Kinda messy and painful to make a run to the bank over and over. Another option is to trade backups with family. You hold a USB drive of their backup and they hold one for you. That works, but still the same problem as the bank, just not as bad.

A much more realistic, and hands-off, approach is “cloud” storage. There are several companies offering secure cloud backup services, the one I chose with is CrashPlan. I went with their “Unlimited” plan, which is about $60 per year. However they are currently offering a 20% discount for new subscribers.

I installed the product on my computer, and configured which directories I wanted backed up. This took a long, long time. I was limited by my bandwidth, and I have LOTS of files (about 3 TB). When I say a long time … it was measured in months, not hours. But once it was complete, a simple backup refresh only takes a few minutes.

CrashPlan will send you a hard drive that you can fill and send back to get your first backup started off quick. Since I was already using another system for off-site backups, I didn’t pay the additional money for that service. However, if my house burns down, I will likely pay for their rapid return service, where they send you a copy of your data on a hard drive if you need it in a hurry.

So you may ask, what if CrashPlan goes out of business? That’s about the same risk exposure as a failed hard drive. Since I have three copies, losing one copy means I have work to do to replace it, but not a tragedy. If CrashPlan goes out of business, or prices themselves out of my budget, then there are plenty of competitors I can switch to.

STEP FIVE – Productions

So now I have my three backups of all my files. However there are auxiliary and less obvious backups. For example, several of the places I upload photos to have them printed will keep files available for some period of time. Also, I have almost every image I’ve produced on my Zenfolio web site, MEDIA.24Moves.com. Likewise, I have fairly large versions of my images stored on Flickr as well.

Now this isn’t all of my work, just the “good stuff” that was worth displaying. Just like you I have LOTS of other stuff from shoots that should probably be deleted rather than backed up. If I lost those source files, it would be sad, but having the productions available on Zenfolio and Flickr is a nice bit of extra security.

I do *NOT* get any warm feelings about the security or safety of using Facebook as a backup system. When you upload images to facebook, their image hosting software takes great liberty with any image that seems “too big”. This includes down-sampling file quality and even reducing the file dimensions in some cases.

End of braindump. What’s your system?

See something here that you might be able to improve? Or do you have a system that works better for you? Anything here helpful enough that it changes your workflow for backups? Please leave a comment and encourage my good behavior.

Joe Lippeatt is a User Experience Engineer, Front End Specialist and Application Developer for 24Moves Consulting. He is also the organizer of HoustonPhotowalks.com Photography Club. When not working, he’s enjoying planning photography trips and helping his wife and son work in their gardens.

How To Fix Broken Google Webmaster Tools for Meetup.com

How To Fix Broken Google Webmaster Tools for Meetup.com

For years, meetup.com let us use “custom domains”. As part of their rebranding and refocus efforts, they are doing away with this feature. Fortunately they are grandfathering (kinda) those of us who were smart enough to brand our groups with a unique URL. But this change has broken access to Google’s Webmaster Tools.

The Issue

And by Kinda Grandfathering, I mean that Meetup.com will accomodate those who already have a domain pointing at meetup.com, but then submit a redirect to your group on their domain. In other words, if you own “houstonphotowalks.com” and point it at meetup before June of 2013, everything worked pretty snazzy. After June ’13, your group members started being redirected to http://www.meetup.com/HoustonPhotowalks/.

Ok, as if that wasn’t bad enough, for those of you tracking your branding and traffic using Webmaster tools, you suddenly realize Webmaster Tools says you don’t own the domain. The reason is your domain cannot be Verified with the Google Webmaster Tools. And to add yet more insult, a lot of the Webmaster Tools aren’t going to be useful because any “houstonphotowalk.com” traffic is automatically shifted to “meetup.com/HoustonPhotowalks”. [sigh] (One note, so far Google Analytics will still work as long as Meetup doesn’t disable those too.)

Workaround

Google will also search your Zone File for TXT entries when trying to verify a domain. If your domain management software (I use godaddy) allows you direct access to edit the Zone File, then open the Webmaster Tools, go to Verification methods, and select Alternative Methods. Simply add a TXT entry with the google-generated TXT Value and pop, you’re verified!

Let me know in comments if you found this helpful!

Safer Lightroom Catalog Backups

Safer Lightroom Catalog Backups

Backups are crazy important. No matter how “good” hardware gets, nothing is 100% reliable. So everyone should have two backup strategies — a backup medium and an off-line backup (even if its just a USB drive you keep in your desk at work!).

Workflows and Backups

Everyone’s workflow is a little different. For me, everything ends up on the Drobo (everything). I even render and work with live files directly from my Drobo. Frequent backups are made to a USB-attached hard drive, and then there are nightly on-line backups for safety.

My workflow pretty much doesn’t use my iMac or MacBook Pro’s hard drives. Everything is on the Drobo. This way, when my iMac dies for good, its no big deal, I haven’t lost my work, just my shirt (having to buy another iMac).

Exceptions to the Rule

One thing that doesn’t follow this workflow is my Lightroom Catalog. Although its perfectly fine working with my images right from the Drobo, for some reason, Lightroom goes into “Walk Like A Snail” mode unless the catalogue file is on a local drive.

So that’s free tip #1: make sure your Lightroom Catalog is on a “local” drive, not USB, not Firewire.

But that’s a bit of a risk. A suddenly dead drive means bu-bye catalogs. Sure, all my images are safe, but loosing the catalogs would be a serious drag.

Lightroom prompts you to do occasionally do “backups” — but the default directory points to the SAME device as your catalog! Sure, that’s helpful if a catalogue is corrupted, but not much help if your drive goes south.

Moving your Default Backup Directory

The good news is, Lightroom lets you change the location of your backups. You can save them just about anywhere you have drive access, including USB attached 2nd backup devices, dropbox, etc.

One drawback is that this setting is in the “Catalog Settings”, not the “General Settings”. So each time you create a new catalog, you will have to follow the following steps. For those of you that create a new catalogue on January 1st each year, that’s gonna be a little hard to remember; might want to set a reminder. 🙂

The Easy Steps

Here are the steps for moving your default Catalog Directory to a different hard drive:

  1. While Lightroom is open, go to Catalog Settings. (Note, NOT the application preferences, this is a catalog setting!)
  2. Under the “General” tab, look for the “Backup” heading, and “Back up catalog” subheadings.
  3. We want to select the “Every time” option. Depending on which Lightroom you use, it may say “Every time Lightroom starts” or “Every time Lightroom exits“. Regardless of which one you have, pick that one.
  4. Restart Lightroom.
  5. At some point, you will see a “Back Up Catalog” popup window, and the first option displayed is the absolute path to the backup directory location. Click the “Choose…” button. You can now select a location on any other drive to store your Lightroom backup catalogs!

This setting persists for each catalog, so you only have to set it once, which is nice. And I suggest using a very descriptive folder name so you don’t mistaken your “backups” for your “actual catalog” (mine are all in a directory called /LightroomCatalogBackups/, clever enough).

Hope this tip helps!

Using an iPhone / iPad for Document Storage

Using an iPhone / iPad for Document Storage

The iBooks App

Many folks know that their iPhone and iPad can read books using the iBook App from the App Store. Its free and very simple to install. What many folks don’t know is, it can be used to store things other than just ebooks.

My iPad is full of PDFs I’ve pulled from the web, mostly application development ebooks and camera gear manuals. A simple google search can find you a manual for just about any device or software on the market.

Installing iBooks

iBooks is a very simple install. Open the App Store and search, its that easy. Its free and works on both iPhone and iPad.

Adding PDF Files from the Web

While in Safari, locate a PDF file you want to save and view it. Usually this is after searching on google and it taking you to a page with a link.

While the PDF is open on your device, click at the very top of the screen, just below the URL and Search bar (see screenshot right).

After tapping in that area, you will get a dark menu bar, with an option for Send To, click that option.

Next you will be asked where to send the PDF document, select iBooks. iBooks will open and the document will be saved to the PDF collection (you can move it later).

Adding PDF Files from Email

If you have a PDF on your desktop, or one that someone has sent you, you can send it directly to iBooks from the Mail app. This works exactly the same way as saving from the web, however your PDF may not automatically download when you retrieve your files. It helps to have a fast internet connection (wifi).

Tap the PDF attachment in your email, it will either open, or start downloading and then open. After the PDF opens, look for the Send To icon at the top right side of the screen; the icon is a little box with an arrow. Tap that icon and select “Open in iBooks”.

Adding Content that is not already PDF Files

This can be tricky, or simple, depending on your set up. If yo have a Mac, saving a file to PDF is really simple. You save the content (Text file, Image, etc) to PDF using the Print function. Then as described above, email the file to yourself and open it on your phone.

You can “screen shot” content on your phone and convert that to PDF as well. For example, if you are on Google maps, screenshot the area you want to keep (but not keep searching for). Then edit that file with any one of several PDF conversion utilities.

Then you can import that directly into iBooks as a PDF

Arranging Documents in iBooks

Once your PDFs are in the iBooks PDFs collection, you can move them around, including creating Custom Collections. In this example, I’m saving all of my travel, shoot info and some maps for my photography trip to Alaska.

In the end, I have all of my documents and maps in an easy to access location, and can even share the Collection with my iPad via wifi for a larger view or share with others.

My Drobo is About To Die! (But That’s Ok)

My Drobo is About To Die! (But That’s Ok)

Recently, Scott Kelby posted his angst against with the Drobo Storage Device, concluding that he’s “Done with Drobo”. His post generated a pretty lively group of comments, including some that disagreed with his statements.

Scott is a great guy, he’s done much for both the design and photographic community. His books are excellent tools, well written and full of beautiful photography and illustrations; I own three.

[EDIT and disclaimer: Its been a while since I’ve bought a Kelby book. Scott and Ben sends me one book a year as part of my work with the Worldwide Photowalk. However, I do recommend them when asked about reading material.]

Unlike some public personalities, he is a really nice guy when you meet him outside his branded persona. And let’s be honest, any person that loves P.F. Changs as much as I do HAS to be a great guy, right? I mean, they were among the first to take the Gluten Free menu to a whole new level.

But I digress.

The Fan’s Respond

Scott’s “I’m Done With Drobo” post resulted in a long thread of replies, which is the best possible result. Its a wealth of excellent alternatives to Drobo for anyone considering a backup system purchase.

However, among the Kelby-fans, there are some who disagree with the premise of the post. I was one of them, and that generated “didn’t you read the article?” replies.

I like my Drobo, but …

First, I’m a Drobo owner, but not a fanboy. There are key areas of Drobo ownership that hasn’t met my expectations. For one thing, the fans are loud. Its hard to hear my wife vacuum (ok, maybe not that loud). This wasn’t apparent in videos posted on Drobo’s web site like they are when I make training videos for work. Also, the startup time double’s my boot time.

Lastly, when swapping drives, it takes nearly a full day for the device to become stable. That’s a full day of work that’s not being backed up (yikes). Good thing is, that’s only been about 4 times a year over the past 3 years.

So far I haven’t had a bricking, but I figure it’s right around the corner (more on that in a minute).

Some Folks Disagree with Kelby

Drobo ownership isn’t why myself and several others disagreed with Scott Kelby’s article. In fact, every one of those who disagreed didn’t take issue with Kelby’s beef with Drobo, rather his assertions about his photographs being “held hostage”.

One of the primary concerns about a Drobo (or similar) device is the proprietary nature of the software. That’s a valid concern. Its important to know that in the event of a failure, you still have access to your data. Since you can’t take your drives out of a Drobo and put them into some other device, this can be an issue. For this reason, some will opt for RAID configurations that are standardized and well-understood.

In fact, that was my 2nd reason for not wanting to buy a Drobo (the first being, they didn’t originally have FW800). When discussing the purchasing options with a Drobo Representative, that was a very specific question I asked.

Backups are Supremely Critical

Honestly, I was not worried that the machine would die; all things that consume electricity have a finite lifespan. But what if it died and the company had gone out of business, how do I get my data back?

The person on the phone was pleasant even though I just suggested the new company she worked for could possibly be doomed. But her reply was stunningly obvious: “Drobo wouldn’t be the only place you store the data.”

… oh. Ever ask a question and then feel really embarrassed when you hear the answer?

Some would say, if it doesn’t exist in 3 places, it doesn’t exist at all. Probably sound advice.

Be Honest about Why You Dislike Drobo Devices

If Scott feels the product no longer fits with his backup strategy, then its important he migrate to something different. If he wants to share the reasoning with his followers, his review of the product may be helpful to others considering a purchase.

However, there were some omissions and areas of the article that could lead people to the wrong conclusions. Its not clear if these omissions were intentional, or would have been corrected if he hadn’t accidentally prematurely posted.

His readers have a high expectation of transparency and honesty, regardless of what product he is reviewing. This posting didn’t reflect Scott’s typical, well-prepared blog post.

The Required Clarifications

So to provide context to my original comment on his blog, here’s my beef with Kelby’s post:

  • He’s been using Drobo for about four years. Four years is probably a bit premature, but all electronic devices fail over time.
  • He own(s/ed) multiple Drobo’s. If each died after 4 years of continuous daily use, it significantly changes the impression left by the article. The article could be interpreted to say he’s had to replace the same Drobo 4 times. Clarification is needed.
  • He insists his award-winning photography collection is being held hostage by Drobo. In fact, he suggests the evil Tech Support people want to extort $100. This is probably the most inflammatory (and unnecessary) portion of his post. The Drobo device should not (and I sincerely hope, is not) the only location he is storing his collection.
  • If Brad is only stored the collection on a single Drobo device, Brad needs to be fired.
  • Drobo’s software is a closed, proprietary system. This may be a perfectly reasonable concern for someone making purchasing decisions. However a company that protects their long-term business health by locking their software in Intellectual Property does not make them evil.
  • Photographers complain that Chinese companies are knocking-off hardware and running American companies out of business — and then complain that companies don’t publish their software’s code. Can’t have it both ways.

Hearing and reading the experiences of actual product users is very helpful when making purchasing decisions. And when well-known, well-trusted individuals endorse (or not) specific products, there is an expectation of transparency and honesty.

The lack of specifics on the previously failed Drobos and the assertion that his images are “held hostage” by the Extortionist Drobo Tech Staff can lead trusting followers to conclusions that are quite likely not true.

*MY* (Honest) Conclusion

My Drobo is pushing 4 years old. When it fails (not if), the time to replace it (possibly with something different) and re-copy everything from my backups would be very disruptive. But I have my backups and in some cases, my original source media.

Backups ensure I will not be held hostage by a bricked, proprietary Drobo system, and neither is Scott Kelby.