Canister

Making it a breeze to work with LTO

Today, we’re launching Canister.

Canister is our LTO backup app for macOS. After a year and a half of development and real-world usage with selected customers, we hear back that we have made something special.

Canister finally makes it easy to work with tape, without getting bothered by all of LTO’s issues.

Get Canister here 👉 hedge.video/canister

Tapes are not hard drives

External hard drives are quite user-friendly — everybody knows how to use one. They’re designed in such a way that it’s hard to screw up, not in the least thanks to USB & Thunderbolt.

LTO, on the other hand, is deeply anchored in the IT department. It’s rock solid tech, but also rocket science if you’re not very computer savvy. Companies like mLogic do great work making LTO hardware easy to use, but with software, life is not that simple. That’s why we built Canister.

Next to everything that you’re used to from us, like a super simple interface, duplicate detection, xxHASH64 validation, and creation of MHLs and Transfer Logs, Canister has a lot of new technology onboard to make your LTO experience frictionless.

Pitfalls

LTO is a beautiful medium, seeing rapid development, with IBM and Fujifilm putting in a lot of R&D to ramp up speed and capacity. Nonetheless, LTO does have its pitfalls — things to consider when working with the medium.

Working with LTO so far hasn’t been straightforward: you need a tape drive, quite possibly a SAS to Thunderbolt adapter (an HBA), LTO tapes, install several frameworks on your Mac, and have a firm knowledge of the do’s and don’ts of LTO tapes to make sure you don’t damage your tapes. Plus, you need a piece of software to manage your backups, unless you like to resort to Terminal for your transfer needs.

Far from ideal, right?

Canister aims to tackle those pitfalls, one by one. Let’s run through them:

Shoeshining

Just like your cassette deck had a play/record head, so does an LTO drive. Each time the head touches down on or lifts up from, the tape, a little wear is introduced. On an audio cassette, this causes noise levels to rise. With LTO, something more important is at stake: data integrity.

To prevent wearing out a tape, reading and writing ideally happen in long, continuous runs. That’s why a LTO drive has an internal buffer of a few hundred MBs, and it will write that buffer in a burst to the tape. If the buffer isn’t filled completely, the tape overshoots and has to lift the head, rewind, drop down, and do the next (partial) burst. Imagine you have a source that can’t keep up with the speed LTO desires. A lot of unwanted winding forth and back will be the result. It’s called shoeshining.

Canister eliminates shoeshining entirely by actively managing the supply and demand between tape and storage. It even works if your storage is a slow network drive. We dubbed it Shoeshine Protection.

Drag ’n’ drop

Because inside a hard drive is a stack of round discs, it’s fast to jump between different sectors of data. This makes it possible to browse through a drive’s contents with no apparent delay. A tape, however, needs to rewind and fast forward between each sector. It’s linear. As a cartridge can hold about 1 KM of tape, zipping forth and back can take quite a while.

To be able to know what is stored on a tape, LTO uses a software counterpart called LTFS — Linear Tape File System. LTFS splits a tape into two partitions, one with the index and one with the actual file data. Having an index makes it possible to browse a tape in Finder. You could even drag files to the tape folder on the Mac, and they would be written to tape.

Unfortunately, doing this will cause the tape to do wear out very fast. Finder will access the data partition even when it only needs to retrieve file information stored in the index partition. It’s a bit weird that this is the major selling point for LTFS touted by the likes of IBM — it is actually the worst advice possible. You do not want to browse a tape with Finder. Ever.

Much better is it to browse a tape without accessing it at all. Canister achieves this by temporarily recreating the index on your computer after mounting a tape. This allows you to cherrypick what you need to retrieve, or create folder structures on a tape before archiving onto it. Only when you’re ready for it, by starting a transfer, Canister accesses and updates the tape. This way that 30-year lifespan isn’t an empty promise.

Drivers

Speaking of LTFS — it’s not just a matter of installing a driver and be done with it. Get ready to have your mind blown 🤯

A new generation of LTO is released every few years or so. With each generation speed and/or capacity are improved. The first LTO generation to support LTFS was LTO-5, and in 2017 LTO-8 was launched.

Since LTO-7, things have become simple: there is one company left that builds LTO drives — IBM. But not everyone uses LTO-7 or higher. LTO drives die hard, so a lot of older ones are put to good use on a daily basis.

For those older generations, LTO life is complicated. There used to be three companies building drives: HP, Quantum and of course IBM. To make an LTO drive work, you need to install drivers, or frameworks on your Mac. The LTFS framework itself relies on another framework called ICU, and it utilizes FUSE to create a virtual file system — like Codex does.

Now, all three vendors have their own version of LTFS and require different versions of ICU to work. One cannot simply install all of them, as the last installed LTFS version overrules earlier ones, but the first ICU framework installed overrules later installs. Mayhem!

Last but not least, there are of course a lot more brands selling LTO drives than the Big Three. There’s no easy way of knowing which vendor made the internals of your drive without digging into System Profiler or Terminal.

This big driver mess is one of the things we wanted to tackle most. Canister comes with a simple installer which allows you to install the set of drivers you need. Not sure which one? Just open Canister, and it will tell you which drivers are missing.

We also made it easy to swap drivers. Keep the installer at hand, and select a new set. Installing those will clean up the existing drivers first so you can be sure your system is set up as it should.

Illegal characters

Another easy trap to fall into is LTO’s character set. As LTO has to work across a wide span of operating systems, file systems, and of course languages, it has a strict set of illegal characters. One way to get around that is to rename files and keep track of what has been renamed into what with a database.

That means that to correctly retrieve media without relinking issues in your NLE, you require a database. Now fast forward to ten years from now, and your LTO software might be way beyond its end of life. With a potential shelf life of 30 years for your tapes, who knows what computing looks like by then? Can you imagine having to support a database from the early 80s, today?

So, preferably no database. Besides that, working around illegal characters is done all the time without needing a database, and you even know how: a space in a URL becomes %24. That’s called URL encoding, and it’s very robust. Each character has a unique code, and there’s zero overhead when encoding or decoding.

Canister utilizes URL encoding for all illegal characters, renaming files on the way to tape, and reversing the process when retrieving from tape. Without the need for a database, there is simply less to worry about — and you’ll know for sure it will work forever.

To index or not to index

When writing data to a hard disk, its partition table, the index of what goes where, is updated on the fly. Not something you ever think about. 
Again, tape behaves very differently. As said before, LTFS divides a tape into two partitions: one for data, and one for the index. After writing data to tape, the index has to be updated. There are three options: after every file has been written, after every x seconds, or when ejecting (“unmount”).

Updating the index for each file is a terrible idea when using frame-based codecs or backing up complete projects with tiny files inside. Indexing every so X minutes is much better, but will interrupt large transfers leading to shoeshining. Worse, the last bit of data written to tape is potentially not indexed if you shut down the tape machine too soon. Index on unmount indexes when, well, unmounting.

The ideal situation would be a mix of all three depending on file size, time or GBs transferred since the last index, and to always finish up when ejecting. Alas, only one can be chosen. It’s a severe limitation of how LTFS was designed — one we’re hoping will be taken care of in a future version of the framework maintained by IBM.

Creating the index on unmount is the only foolproof mechanism to make sure what has been written to tape is safe and sound. That’s why Canister ejects each tape when a transfer to tape is done. Luckily, mechanically reingesting the tape is also supported by LTO drives, so you don’t have to go and insert a tape when remounting.

Looking forward

We think Canister will make your LTO life far easier. But there is always a need for more features, for more workflows. That’s why we’re giving you a voice in our roadmap. Which feature do you think we should build first?

• A database, to browse your tapes without having to mount them.
• Spanning of large transfers across multiple tapes
• Migrate old tapes to newer tape generations
• Tape library support
• Conform

Available now

For a limited time, Canister is available not for $199, but for just $149. Get it at hedge.video/canister where you can also download a free 10 day trial.

If you have any questions, please let us know through canister@hedge.video or our Twitter and Facebook pages.