Postlab Drive

A journey into shared storage, in the cloud.

When we re-released Postlab in late 2019, having converted it from an on-premise tool to a cloud-driven app, it became instantly clear what the missing link was: media sharing.

Now that editors can share and collaborate directly in NLE projects, and the whole world is forced to work from home, overnight all you suddenly needed to shuttle your media and all others assets - and not by using old fashioned SneakerNet™ 👟

Partly cloudy...

We don't like building that what already exists. At first we tried to help Postlab teams with advice on how to best set up their existing cloud storage like Dropbox, Box, OneDrive, and Google Drive, for a remote editing workflow.

But no matter how hard our workflow architects Felipe, Isaac T, and Felix tried, customers kept being borked by invisible barriers: erratic upload and/or download speeds, file size limitations, unpredictable latency, and non-public data caps that incur a speed penalty after x GBs of downloads per day. Cool for Word documents, but not for media workflows.

The only way to make "prosumer" clouds work reliably for video production turned out to be cloning your cloud drive in full to a local disk, and keep it in sync with a tool like rsync. In reality, this results in heavy overnight syncing, which is far from real-time collaboration. Add to that the vanishing of creation and modification dates, and illegal characters that make no sense, and you end up working around a giant snakepit of exceptions.

...with chance of rain.

Some customers, in the meantime, switched to using "professional" clouds, i.e. object storage or S3-compliant storage. This solves the speed and reliability issues, but also comes with a hefty and unpredictable price tag. For starters, you pay for every bit that's downloaded ("egress fees"), and often as well for deletions, file properties lookups, plus a premium for picking the server location that's nearest to you.

Cost of ownership is important in a strict budgetary environment like video production, so seeing your bill explode is never OK. Apart from that, treating raw object storage as a file system doesn't exactly improve your data's integrity. It just wasn't built for that. Also, not all object storage is hot storage: it may take minutes and even up to a day to retrieve your files if you decided to go for a cheaper storage tier. So, off-the-shelf cloud object storage is also not a solution unless you have deep pockets to cover for the hidden cost.

One more thing also became apparent: a large local cache of the downloaded data is mandatory. Redownloading all files each time is not an option in a world where 1GB connectivity is not ubiquitous.

Clear skies

Then it struck us: you don't want cloud storage per se; you want to work remotely as if you're in the office. As if you're on shared storage. You need remote shared storage, that behaves like it's local, something that Finder can work with, and that has a clear price tag:

• Built for media
• Reliable speed, low latency
• Behaves like local storage
• Predictable pricing

Thus, time to reinvent the wheel 😁

We wanted to build a cloud drive that acts as if it's a hard drive sitting next to you on your desk. We also realized we shouldn't start from scratch but rather work together with partners to combine existing technologies into one-cloud-drive-to-rule-them-all.

Meet Postlab Drive 🚀

When Air Masses Collide

Shared storage comes in two flavors: NAS and SAN. The former uses network protocols to give you file-based access, whereas the latter gives you block-based access by posing as a local filesystem. Network protocols have a lot of overhead, so a SAN is often the better choice when you need speed - albeit costly due to the upfront investment and use of Fibre Channel.

So why not put a SAN in the cloud? Well, AWS does precisely that. Their Elastic Block Storage product range gives you a SAN in the cloud – starting at $150 per TB per month plus egress fees, which tend to be a multiple of storage fees. As with any SAN, it needs an initiator: software that mounts the storage and translates the blocks into files. With AWS, that initiator can only be run on an EC2 instance in the same cloud as the EBS instance. This means you can't access your SAN directly from anywhere else. Going through the EC2 instance to get data to your Mac is possible, but it would obliterate the speed advantage. Apart from all of that, it performs very well. If you need to do VFX renders, be sure to investigate running an EC2+EBS render farm. Other than that, EBS only checks one of our boxes: speed.

That's why cloud vendors typically use object storage. It's cheap and convenient: inexpensive hardware, no overhead from network protocols, and you can store whatever you want inside an object. Object storage is great for archiving, and thus, pricing is adjusted to that - it's fairly cheap to store data, but not so cheap to download, or utilize the data in the cloud.

Interfacing with object storage is done only through APIs, and that's not how you work: you need something that Finder understands – a file system. For software,  though, it's great to have an API, so we're onto something here. Again, only one box is checked - partially: price, without predictability.

Ideally, we want block-based speed and object-based ease of use and price. Chopping up files into blocks and storing those on object storage isn't hard to do. Still, without the aforementioned initiator, the engine that recombines blocks into files, it's useless - plus a cache controller, because you don't want to keep redownloading media.

Altocumulus

So, to build a cloud drive specifically for media production, we needed multiple moving parts: block-based storage with multiple worldwide locations, an initiator, a cache controller, access management, and a very easy way for non-customers to move data into and out of the cloud without needing a subscription. Quite the list.

First stop: pricing. We wanted Drive to be a flat fee product. That means we needed a cloud vendor that did not have egress cost as its business model. That typically means one that aims at semi-archiving media, or Tier 3 cloud storage - but with a backend that is built to perform like it's Tier 2 like AWS has. A backend that is built to scale.

Enter Wasabi - they focus on Tier 3-type customers, but have an architecture that is more like Tier 2 (for Tier 1 and 0, think EBS). By working with Wasabi, we can offer truly low latency, no download caps, no minimum storage period, no fair use policy, and no speed limitations. That checks a lot of boxes ☑️

Next stop: a file system that's fast, block-based, and unobtrusive. Enter Filespaces: a technology of Lucidlink, that builds on top of object storage, and lets your computer act as the initiator. For the techies - it's like StorNext, but in the cloud. We've been friends with LucidLink for over five years, and we immediately thought of them when we realized we were in need of a good file system. But a file system by itself isn't enough - you need to actively administer and manage it. Luckily, we already knew how to do that 😁

Postlab manages all technical bits for you: credentials, configuration, updates, security, cache management, drivers, and more. You'll only need Finder.

All of that culminates into something pretty fast:

When it comes to doing a roundtrip of uploading and download a clip, Drive is on par with Frame.io (who offer an amazing upload, by the way), about twice as fast as Google Drive, and about 3x as fast as Dropbox and OneDrive.

In reality, you won't even have to wait this "long" before a clip has downloaded - as soon as the first few MBs have been downloaded, you can play back a clip. That's what makes Drive truly unique.

So, that's all boxes checked — except one:

Drive Thru

Getting data in and out of the cloud is maybe even more important than being able to work with the data when it has been uploaded. As a Postlab Drive user, you can simply drag files into Drive with Finder, and they'll start uploading immediately. But, that's just you and your team.

For your customers it's always a lot more hassle. They always need help, or need to get an account, get a subscription, or some other kind of time sucking friction that turns you into a media manager instead of an editor. And when the customer has finally managed to upload assets, you still need to spend time on finding it, and moving it to a location where you can work with it.

That's why Drive has a fourth pillar: Drive Thru. It's our own WeTransfer, if you will, and it's a two-way street:

With Drop Off, you bookmark a folder on Drive and Postlab creates a direct link to it, to give to your client. Uploads go straight into that folder - right where your NLE expects them to be. You won't even have to leave your NLE to start editing.

Pick Up is the opposite: pick a folder on Drive, and send your customer a direct link to download its contents. Made a change in the mean time? The customer always gets the latest version.

If you or your customer have too much data to upload, you can also send us your storage devices and have us do the heavy lifting, for free.

Kyno, MASV, and iconik

No one works with just one tool, so we feel integrating with other tools is paramount to making cloud life better - otherwise a cloud drive is just a drive hooked up to another computer, somewhere in a datacenter. There's no fun in that.

We're kicking off Drive integrations with three great partners:

MASV, iconik, and Kyno.

Kyno's Drill Down feature on the desktop lets you, well, drill down a Drive and detect all media that's buried deep inside. Drill Down for Postlab Drive ups the ante: it fetches metadata, creates thumbnails, and creates the sidecars Kyno uses - all before you even open Kyno on your desktop. Have a DIT upload media to Drive on-set with Drop Off, and as soon as your AE opens Kyno to tag and process the media, it's all already there.

MASV helps out in those cases where you need to get data into the hands of editors as fast as possible. Set up a Postlab Portal in MASV, and you can have anyone upload media straight to Drive. But wait? Doesn't that sounds like Drop Off? It does.

The key difference is that Drop Off always uploads to directly into Drive - meaning that if Postlab Drive is on our SF servers, and you're for instance in LA, it will be fast. But if you're in Amsterdam, it obviously will take longer. With MASV, not so much: you'll upload to the MASV server that is physically closest to you, and then MASV picks an accelarated path to your Drive server location. Our benchmarks show it's at least twice as fast, so that should help a lot if you're shooting in the middle of nowhere and still need to get clips to editorial.

Our integration with iconik is similar to Kyno's: it turns Drive into an Iconik Storage Gateway. When media gets uploaded to Drive, Drive will convert your media into proxies for use in iconik. Supercharge that workflow by using Hedge to upload media to Drive: paired with Hedge's Source Review and iconik license add-ons you ingest your media complete with metadata that becomes immediately  available in iconik.

Security

Postlab Drive offers extensive security, on several levels. We've built a tiered security system, consisting of multiple moving parts. Like in a nuclear submarine, only when all credentials are present and validated, access it granted.

• Your Postlab Drive credentials are separate from your Postlab credentials, and automatically renewed on a more-than-daily basis. Your Postlab credentials work as decryption keys for your team's Drive credentials, but not before Postlab checks if you actually still have access. Even better, you don't have direct access to your Drive credentials - you always have to log into Postlab first, to reach the next layer. Cobb says hi 👋
• LucidLink's "Zero Knowledge" encryption model ensures that without access to a file's metadata, the blocks are useless blobs of data.
• Wasabi's SOC-2 and ISO 27001, and IAM policies.
• Bring your own S3 - instead of using our storage backend, bring your own. You foot the bill for the storage and only pay a little premium for Drive's functionality.  

We take security pretty serious, so if you have any questions about it, please reach out.

A Journey Into Clouds

That was a six-month process, in a nutshell. Drive is available as of today, for all Postlab users, and also for non-Postlab users. Even if Postlab's collaboration tools are not what you need, Drive will still be of help in many ways - sharing data between people is just one. And in case you do end up needing Postlab, it's already included with Drive - everyone that just needs Drive, also gets access to Postlab Solo. The best of both worlds.