This will be a bit more technical than normal.  This is also the post that has taken me the longest to get correct.  I think I started this this post 4 years ago.

I have a lot of photos.  My first digital point and shoot was a Christmas present nearly 25 years ago.  Around 15 years ago, I had most of my film scanned.  This means I have around 240,000 photos and videos consuming nearly 10 TB of drive space.  Managing, and even copying, all of this data is quite a challenge.

Here is how I handle this.

Storing 10TB

I use 4 drives to store all of my files. The internal drive on my computer is by far the fastest, and so it becomes the landing spot for everything. After initial sorting, tagging, etc., I move to another internal drive, this time an 8 TB traditional internal drive.  This drive has most of 2022 and all of 2023, but it is only a few years old, so it will probably eventually hold the last 4-5 years.

Internal drives are always faster than a comparable external drive.  I’m only using about 1 TB of this drive right now.  It took over the responsibilities of my 4 TB USB drive.

After a few years, I move things to my archive disk.  This drive is an 8 TB external drive that is now full.  This drive holds, as I just noticed, 230,000 or my 241,000 images. I probably shouldn’t do that.

I do have a semi-retired 4 TB external USB drive.  This drive was probably purchased in 2012-2014 and previously held both roles that are now handled by the 8 TB drives.  It isn’t retired, but the job it performed has been taken over.  I might replace the drive but keep the housing, it is quite nice.

As my main computer is a desktop with about a dozen USB ports, these drives are always connected.

If I need to do a lot of work on a group of older files, I will sometimes move them back to an internal drive for a few weeks, then back to the external drive when work is finished.

In the early days, my digital camera files were very small.  My first camera was 1.3 MPix and only shot in jpg.  Every photo I shot on this camera will fit on 1-2 burned CDs.  My second digital camera was 3.0 MPix and also was jpg only, but was in use for a longer period of time.  Every photo and video from this camera will fit on 1-2 DVDs.

Even considering that my first DSLR, a Nikon D70 in 2004 and then a D300 in 2009, every photo from 1999-2010 consumes around 283 GB.

 

2014 – 2022 were each at least double the total of 1999-2010. 2019 is unique, but it alone was more than 1 TB.  2020 and 2021 were 700-850 GB.

I have broken down storage in my previous ‘Year in Review‘ posts.

Backing up

2 is 1 and 1 is none.

This means if you don’t have at least 2 copies of your data, you don’t have your data.

The best advice is to follow the 3-2-1 rule, which is usually explained as something like: 3 copies on at least 2 types of media with 1 offsite backup.

For my offsite, I use Backblaze (sign up link with my referral) and have been using them since at least 2012.  They really saved me back in 2017 when my Mac died. In this Mac, and my internal drive and one external drive had full disk encryption turned on.

When returning from a big trip, I usually wait for Backblaze to finish before doing a lot of editing work.  This can take quite a while.  For example, when I returned from Greenland with 455 GB of new photos and videos, it took a few days to upload all of the files to Backblaze. When I did my initial upload, I was on very fast internet and still took a few weeks. When I did my first upload back in 2012, it took several months. Realistically, the max upload I’ve ever managed on Backblaze is 250-300 GB per day, even on 1 Gbit upload.  So in an ideal situation, 10 TB would take 30-40 days to upload again.  From experience, Backblaze uploads will not saturate a 1000Mbit connection.  300Mbit upload service will be just as fast as 1000.

I also have a local NAS from Synology.  This device has 5 drives at 4 TB each. After formatting and setting up RAID, I have 14 TB of usable space. The NAS can even lose a drive and continue to function, with all data being available but slower to access.

Moving The Data

Having a plan for the data is great. But now remember that to back up online, you need to be able to upload several TB of data. You’ll also want to do this over a reasonable period of time. To copy to a NAS device, you’ll need to be able to copy several TB of data over my internal network.

Both of these start with a home network and gigabit, or higher, ethernet. You will also need to ensure that your computer is connecting to the NAS and the internet by a wire. I have a gigabit hub that connects my computer to my NAS. Copying 5-6 TB with any regularity takes time (I do full backups monthly, partials every day) – given how much Windows 10 likes to sleep, this takes about a day to a day and a half for a full copy over the network.

While the local copy is being created, Backblaze is looking for new files and uploading them to the cloud. I was fortunate enough to have gigabit internet uploads speeds in the past. This makes uploading to the cloud a lot easier.  It is still possible to upload 100’s of GB of data over 50 Mbit internet, it just takes a lot longer.

What about when traveling?

When traveling, I use a 1.5 TB USB drive that does not need external power.  I always ensure that, when importing new photos and before formatting the cards, there is a second copy of the images on this drive.  I usually try to make several full backups of the photos every few days, as well as delta backups daily, during travel.  When the internal drive fills up, I no longer format memory cards.  When a card is full, it is full.

Upon returning home, I plug this drive into my desktop, copy the files into the appropriate folder, and then import into Lightroom.  I use the sidecar files in Lightroom, so, most of the changes I’ve made on the trip are simply brought into my main LR catalogue.

So Does this Work?

Yes.  When it comes to backing up files, paranoia is never a bad plan.

I haven’t lost a file unintentionally since around 2000.  And even then I lost a bunch of files because I formatted a card before copying the files off.  This taught me to copy files immediately, which is part of the reason I keep an external drive around.

Beyond that, I’ve designed the system so that it is difficult to delete a file.  Which is both good and bad.  It does mean that I sometimes find photos that I thought I no longer had and others that I don’t intend to keep. But this is probably better than the alternative.

 

Until Next Time

-Brad