Backup Bottlenecks #2: frames vs clips

This is part 2 in a series about speeding up backups. Read part 1 here.

In part 1 we learned that moving a big file is more efficient than moving a bunch of small files. So what can you do to speed up your workflow?

Archives, bundles and catalogs

Contrary to what many think, storing your clips in a big catalog like FCP’s Camera Archives or photos in a Lightroom Catalog isn’t helping at all: it’s just a folder with your files inside. Easier for you, but not for the disk. However, compressed archives are, the most useful being a Disk Image (DMG).

If your workflow requires you to store clones of media cards, consider creating DMGs straight from the source disks. DMGs are Mac-only disk images, and can be created with Disk Utility. You can also use an automated solution like a Hedge Script to create DMGs alongside regular backups, to store them for long-term archiving or push them straight to LTO. A cross platform alternative for DMG is an ISO.

An alternative to disk images is of course zip or tar, but such a compressed folder is not an exact copy of the source material per se. It contains hidden system files created by Finder and other processes. Only a disk image is a true clone.

If you wish to store archives in the cloud with services like B2 or AWS, a single big file is also cheaper to store and retrieve than a bunch of files, even if the total size is the same. It’s the I/O count that matters on your bill as much as the storage itself.

Frame based versus clip based

We asked Daan Hermans from pro video workflow experts The Future Store about good practices when working with different file sizes:

“ I’d say it’s mostly about whether you work frame based or clip based.

Frame based gives you flexibility but puts stress on your storage: for every frame a disk needs to look up its location, and they are spread out randomly across the disk. SSDs are particularly good at this compared to HDDs because there’s no hardware to move around inside the disk. Thus, it makes sense to invest in Thunderbolt SSDs if this is your workflow, whether you’re offloading or editing.

When working with clips, a disk is accessed sequentially. Once a clip is located the file system knowns where the next read/write will need to take place: directly after the previous one. The larger the file, the more efficient it becomes. Random access is only needed when looking for the next clip.

If you are experiencing a bottleneck when working with frame based files, and don’t have the funds for an SSD, it’s smart to create ProRes clips of your shots and continue working with those.”

No shocking conclusion there: spend money to gain speed. It does show how useful it is to spend money on HDD filled NAS or RAID systems to crank up speed when a SSD might do the job on set just fine. The SSD is more expensive, but the smaller foot print is worth something too:

“ We compared a WD Elements, Freecom Quatro, LaCie Rugged Thunderbolt SSD and finally a G-Technology G-DRIVE ev SSD. We moved 20 GB of tiny files from a Macbook Pro internal SSD to all those disks one at a time, with Hedge.

Using a SSD for offloading saves you at least 15 minutes per copy hour. That easily adds up to a few hours a day. If you use a mobile disk like the Element because of the form factor, you could save up to 46 minutes, every hour, by using the G-Tech SSD. That’s money very, very well spent.”

About using a NAS on set:

“ To us it makes little sense to use a NAS when looking for speed. A NAS is useful for multi-client access, and that’s generally not needed on set. A GigE connection may seems fast, but it probably won’t be fully utilized as the real bottleneck is the underpowered ARM chip inside the NAS. The CPU is very busy with calculating the RAID table and spreading out all the data across multiple disks. The exception of course is a professional 10 GigE NAS like a G-RACK, but the form factor is differs a lot from a desktop NAS.”

On RAID:

“ RAID through Thunderbolt is a different story altogether. Because it is a direct PCI or SATA connection, there’s no overhead lost on network protocols. DITs should use either RAID1 or 5 to be prepared for both frame based and clip based workflows, and still enjoy relative peace of mind because of the added redundancy.

RAID1 doesn’t give you better write speeds, but read performance is multiplied because of simultaneous access. RAID5 eases the stress of writing across all disks, so use that if the type of production demands higher write speeds.

Fill it with SSDs and nothing’s going to stop you. Of course, this is just useful for fast offloading and playing back 4k footage. It’s not necessary for storage of RAW footage or archives. That’s where cheap HDDs still have a place.”

Takeaways

• Consider if your workflow is file or frame based.
• SSDs may still seem expensive, but the time saved on set is huge.
• For offloading, don’t spend money on RAID or NAS filled with HDDs.
• Thunderbolt trumps network access, if you don’t need multi-client access.

Big thanks to Daan & the other guys @ Future Store for the benchmarks!

Liked this? Here’s a $10 coupon for Hedge, the fastest way to back up media.