How to speed up your on-set backups, and save hours.

Paul Matthijs Lombert

• 3 min read
How to speed up your on-set backups, and save hours.

…and save hours.

After a long shoot nobody likes waiting for the backups to finish. It’s not so hard to prevent this from happening. Choosing the right setup can literally save you hours.

To power or not to power

When talking to our users it struck us that there are roughly two camps: the portable bus powered hard drive camp, and the AC powered hard drive camp.

Want speed? Join the latter. Yes, those power bricks kill mobility. Yes, they break or get lost. But they’re around for a reason. Have a look at the speed tests we did:

Every hour spent backing up footage to bus powered disks could’ve been done in 22 minutes less. The price tag for both types of disks is pretty similar, so if you can spare the space and weight it’s worth the hassle.

Hubbies

There’s more to it than just disks. USB3 and Thunderbolt have room to spare: one disk won’t use up all the bandwidth available. This enables you to run simultaneous transfers. Macs don’t come with an abundance of ports so youll need a hub, or two if you separate reading and writing.

We benchmarked a D-Link USB3 hub, that you can use both bus powered and ac-powered:

Shorter is better, times in minutes:seconds

If you need to offload multiple cards simultaneously it’s definitely worth spending around 50 bucks for a good hub.

Simultaneous Transfers

Let’s see if we can speed things up even more. Hedge’s concurrency scales up dynamically with the amount of disks you have in use. The trick here is that while files are being imported, other ones can be verified [1]. This ensures the optimum amount of bandwidth is used.

For this test we used the MD5 algorithm that’s in use by most data handling and backup apps. Hedge itself uses the more modern xxHash algorithm [2] which is at least 15% faster than MD5. With faster media the speed gain is even higher [3].

Check out these benchmarks when using Hedge with and without simultaneous transfers:

Shorter is better, times in minutes:seconds

Pretty conclusive, right? If you import your media with verification, simultaneous transfers are the way to go.

Queue Up

If you got a big shoot, make sure you keep your import queue filled all the time. Don’t wait for a media disk to run full, but swap as often as possible. Still not fast enough? Renting more media disks could end up being a lot cheaper than working overtime.

Best practices

Keeping a few things in mind will save you hours down the road:

  • Use concurrent transfers to make good use of bandwidth.
  • Keep those card readers busy!
  • Use external disks with a power adapter when you can.
  • Don’t use bus powered USB hubs.

Thanks for reading! If you enjoyed it, hit that heart button below so that others can discover it too.

Notes

[1] If you don’t use verification on your files (i.e. when using the Finder) you don’t gain anything from concurrency. If this is your situation, you’ve got a bigger problem: your footage is in Limbo. Always verify your footage!

[2] MD5 tops out at speeds near 300 MB/s. xxHash’s speed is determined by the speed of your RAM and has no ceiling. In practice this comes down to the newer Mac, the faster the verification. MD5 will never improve in speed, and becomes the bottleneck as USB and Thunderbolt become increasingly faster.

[3] We had one user reporting a 68% speedup when offloading a SxS card onto an external SSD. It took him 13 minutes instead of 40 minutes, using xxHash instead of MD5.