Speed 2.0

Paul Matthijs Lombert

• 7 min read
Speed 2.0
Hedge 22.3 - the fastest Hedge, by a landslide. Plus: Finder & Explorer integrations.

Back in 2016, spinning hard disks (HDDs) were the go-to option; large capacity SSDs were still unaffordable to many productions. That's what we consequently built Hedge's copy engines for, to make use of every bit of bandwidth available on your hard drives. Today, HDDs are no longer the standard; non-SATA SSDs and NVMe's are. Time for some engine tuning.

Over the course of a year, we've been working in tandem with our partners Sandisk Pro and iodyne in researching ways to achieve a structural speed jump. A jump that isn't just for those with bleeding-edge NVMe storage and brand-new computers, but one that would benefit everyone.

Hedge 22.3 is the first iteration of this "Speed 2.0" R&D, and focuses on improvements around the storage itself, file systems, and networks.

Gear

There are a lot of test results in this blog, so let's talk for a minute about the gear we used. As the goal was for everyone to benefit, all tests were done with standard, not too modern, gear:

Nothing fancy. Keeps things honest, doesn't it?

Warp Speed

6 years after the release of Hedge 1.0, the storage landscape looks very different: in 2016, USB 3.0 (excuse me, "USB 3.1 gen 1") was brand spanking new, Thunderbolt drives were expensive, and SSDs had tiny capacities - barely worth using for anything other than refitting old Macs with a new system drive to give them a speed bump.

Present day: NVMe drives are everywhere. It's the dawn of a new decade: what came to market 10 years ago is now so mature that it's becoming disruptive - faster, better, and cheaper. New NVMe-based camera cards like CFExpress are superfast, SSDs are nearing the $100/TB price point, and the m.2 form factor allows people to buy empty enclosures, to outfit them with SSDs based on each production's particular need — SSDs finally democratized storage.

As NVMe scales indefinitely, this world needs copy engines that scale along with it. Like you shouldn't be held back by a verification algorithm (MD5 has a 300MB/s limitation, that's why XXHash came to the scene - it scales along with RAM speed), we feel you should be able to use all the speed your storage offers.

We'll spare you the details of the work we did — the new Hedge is at least 20% faster for all types of direct attached storage that Hedge wasn't already maxing out - and often, a lot, lot more.

One On One

Copying from one source to one destination, or 1:1, Hedge already excelled at. However, not all filesystems and operations systems are created equal, and that has an impact. Your copies are only as fast as the file system can get out of the way. Especially exFAT deserves some special attention.

The good

Why use exFAT in the first place? The most cited reason is that you have, or might someday have, a customer that's on Windows while you're on Mac, or vice versa.

True, it's the only filesystem that works on both Windows and macOS. Cross-platform compatibility is a really great feature - when done properly.

The downsides, however, are greater. Ejecting an exFAT drive without properly un-mounting it first can and will kill the partition table. 100% data loss guaranteed.

And there's more:

The bad

ExFAT is a lot slower than other file systems. Even with all of Hedge's new speed gains in place, there's still some overhead left. That's a lot of time lost for the sake of a potential Windows customer.

The SSDs we used for these tests max out at around 1400-1500 MB/s:

This clearly shows the power of Apple silicon - same storage, same connections, but much better performance due to Apple's relentless hardware optimizations - keep in mind that the M1 has 8 GB less RAM.

But what's going on with Ventura?

The Ugly

ExFAT has a major speed regression in macOS Ventura; Hedge performs about 40% less than on previous macOS releases, and it's worse for the other apps in the space - up to 60% speed loss. This is unfortunately something only Apple can fix, and we're in talks with them to find a solution.

If you need to use exFAT, don't update to Ventura.

The Best

The solution to all of this is simple: for your destinations, stay away from exFAT. It's best to use AFPS instead, and give your customers a Paragon for APFS  license to use on Windows. Best way to spend $49, we'd say. It'll save you countless hours.

If your camera or recorder gives you the option to use a different file system besides exFAT, it's likely HFS+; Apple's old file system. That's perfectly fine too - Paragon has a driver for that as well. If it's NTFS, you're in the opposite situation, as it natively works on Windows but requires a Paragon driver on macOS.

One to many

When doing camera offloads, you'll have to create multiple backups. Hedge's focus in 2016 was to ensure there's no overhead when doing multiple transfers from the same source - and we've been able to keep that up.

Even Finder is pretty good at 1:1 but doing one-to-many copies all at the same time is quite difficult as you'll need to have magic hands to start two copies at the same time. Because Finder and other apps don't have the Transfer Sync technology that Hedge has, at some point, read and write operations go out of sync and lead up to a 55% slowdown:

...and it's worse for multiple sources:

Many to many

Backing up multiple camera cards to multiple destinations, all at the same time is where Hedge's power really starts to show. It's the only app that can do it effortlessly. The original Hedge was already so good at it that it often outpaced the competition by 50%. We've managed to squeeze out another 10% on top of the other speed gains.

10% does not sound like much, but as it's a gain per discrete transfer, it quickly compounds when you add more sources and destinations:

In real life, you won't be able to get these speeds with Finder - it will be at least 20% slower as your fingers lack the benefit of the automation we used for these tests.

Archiving

So what does this mean for archival workflows? For starters, Hedge no longer creates MD5s by default when in Archive mode - MD5 is just too old and long in the tooth. Moving MD5, C4, and SHA into a separate option in the preferences removes that bottleneck, and the gains are pretty big when combined with the engine tuning:

Networking

Hedge's network engine was already pretty solid, both for gigabit (1GE) connections, that have been available on most computers for over a decade, as for 10GE, which is quickly becoming the de-facto standard for pro video.

1GE can theoretically sustain 125 MB/s, but that's unrealistic due to the network protocol's overhead. In reality, 85 MB/s is considered by many to be pretty OK. On the Mac, we've been able to dial that up to 110 MB/s — 30% more.

For 10GE on macOS, the gain is even more significant at up to 75%. That's what really starts moving the needle: it means you can ingest 14TB more onto a NAS each day.

On Windows, the network stack is a lot better, so there's more room for improvement: 1GE network transfers are, on average, 70% faster and 10GE a whopping 150%.

With Lumaforge (now part of OWC) leading the 10GE NAS pack, high-speed network transfers are becoming just as ubiquitous as direct storage transfers. It's fair to say that soon each Ethernet port on a new computer will be of the 10GE variety. If your Mac doesn't come with a 10GE adapter, we can highly recommend Sonnet's Thunderbolt 10GE adapter. It's very affordable, driverless, and rock solid.

A Finder replacement

So, with Hedge now on par with Finder - and beyond - why not use Hedge for all copy actions? That's what a bunch of customers have been requesting, and it's now the right time for it.

Hedge now comes with two Finder/Explorer extensions: Copy & Paste for an automatic workflow, and for a more hands-on approach, an extension for setting Sources and Destinations.

Available Now

All these speed gains are available as of today, in Hedge 22.3. Let's summarize:

  • All direct-attached storage that didn't already max out is at least 20% faster.
  • 1GE network transfers are 30% faster on Mac, and 70% on Windows.
  • 10GE gets a 75% bump on Mac, and 150% faster on Windows.
  • Transferring to or from exFAT on Mac is 40% faster.
  • Archive mode is up to 240% faster.

Also, as all these tests were done on not-too-new and shiny computers, your gains are likely even larger if you have a newer machine.

This is just the first step in our Speed 2.0 project. There's a lot more to come 🚀