Catalogs for Canister

Paul Matthijs Lombert

• 5 min read
Catalogs for Canister

Ever looked at a pile of hard drives only to wonder on which one that clip with that particular shot was, and dreading to have to mount them all to find it? Now, envision that with LTO tapes — where mounting them is not a matter of seconds but minutes. To solve that, we've been working on a new technology: Catalogs.

We're bringing it to Canister first 👇

No Databases, Please

Everyone has principles, and one of ours is "no vendor lock-in". As former sound engineers, we've all been locked into a certain DAW for way too long, and that is not something we want our users to experience. That's why for years we've been saying "no" to resorting to databases. Apart from the lock-in, they're a single point of failure, so just like with backups, they will get corrupted sooner or later.

But, when working with LTO, having a database is really useful. It allows you to track all tapes you ever made and more. Nothing is as demotivating as having to go through that stack of LTOs to find that one particular file. So, we needed something like a database, only better. Something that checked these boxes:

  1. Searchable - LTOs typically hold millions of files, so search is a necessity.
  2. Non-destructive — corrupted data should not make the whole inaccessible.
  3. Versioned — all changes should be tracked, not just the last state.
  4. Immutable — it should be not too easy to edit data (but also not impossible.)
  5. Lightweight — databases can have a heavy footprint, we don't want that.
  6. Native  — don't reinvent the wheel, use proven OS technology where possible.
  7. Learning — it should be easy to add existing tapes.

Catalogs is all of that, and some more.

Each time a tape is seen by Canister, a snapshot of its contents is created and stored locally, as a collection of stub files and folders. After archiving, the snapshot is updated.

All these snapshots ("catalogs", you guessed it) are made available as one read-only network drive, configured to be searchable by Finder. The network drive (dubbed "LTO Archive") is automatically made available when Canister starts and is directly accessible in Finder.

Spot the new Catalog button ☝️

Finder is your friend

Many don't realize it, but Finder is the most used app on macOS, and because of that, it's very advanced. It's quite a feat to build something as feature-heavy, robust, and stable as Finder. By utilizing Finder as the UX for Catalogs, a whole slew of opportunities opened up:

  • Smart Collections let you keep track of new additions.
  • We can use identical filenames, no need for confusing extensions like .p5c.
  • If a stub goes corrupt, nothing breaks. Not even the stub, and definitely not the entire catalog.
  • Sharing the Catalogs folder on the actual network, or on Dropbox, makes your archive available to the whole organization.
  • Versioning without any overhead:

Often overlooked, versioning is really important when you're tracking data. All previous versions of a catalog are saved too, in hidden .history folders.

Missing files in a catalog? Most likely cause: someone accessed the tape directly in Finder and deleted or renamed files. Just make your hidden files in Finder visible and check the history catalogs to see what happened when (Pro tip: Shift+Command+.)

Best of all: it's super easy to create catalogs of existing tapes. Just pop them in, mount, and eject. That's all.

Future Proof

Using a file system as the basis for Catalogs has a lot of upsides. Its footprint is tiny; 1 million files come in at just over 1 MB, and it allows for all kinds of additional features. Because the need for Catalogs is the highest for LTO users, we implemented the technology into Canister first. But that's not where we'll stop: Catalogs will come to Hedge, to Foolcat, to Checkpoint, and maybe even to Postlab. Catalogs will become a central concept in all our apps.

We're also thinking about populating Catalogs with metadata created by Foolcat, by Hedge, or by yourself, adding previews. Heck, we could even create mini supercuts of clips and make them scrubbable (is that a word?). Detect clapperboards and add those as a preview image for Quicklook. Right-click on a file in Finder to kick off workflows like pulls and conforms... so many opportunities to make LTO workflow. Does Catalogs tickle your brain? Are you seeing useful ways to utilize or expand it? Let us know, we're all ears.  

Driver Mayhem

One thing we unfortunately had to change in this release, is that we no longer offer to install drivers during installation of Canister. With Catalina's new security protocols (justly) in place, the hammer has finally come down on software vendors that do not sign their LTFS and ICU frameworks. On top of that, FUSE is no longer open-source (but still free for end-users) and now requires a manual installation. The same goes for the new Catalogs: you'll need Apple's Command Line Tools and bindfs, an extension of FUSE.

But fear not - we have come up with a solution that is quite simple: on startup, Canister will detect which drivers your system lacks and offer to download them straight from our servers. Then you come in: if your OS reports it cannot open the installers because the developer cannot be identified, open the packages with a right-click and choose Open. (Yes, that differs from just double-clicking it!). That allows you to install them, which takes just a few seconds.

If you think this is silly, we completely agree. But don't blame Apple. The blame is on IBM, Quantum, and HPE that have been lacking to sign their frameworks properly for a decade. They need to step up their game and properly sign LTFS and ICU. Until then, it's up to you to do some manual work to get started.

(We could get a license for FUSE to install it along with Canister, but it would require us to raise Canister pricing by $100. We feel that is a bit too much to charge you for saving a few seconds of setup time.)

Available as of now

Catalogs are available in Canister 20.2 as of today. It's a free update if you bought Canister less than a year ago and only $99 for older licenses. We've emailed all existing users an upgrade coupon 🥳

For new users, a Canister license is $299, and we now also have a bundle with Hedge for $399. Don't have an LTO drive but want to start archiving for the long term? Get Canister for free with mLogic's great mTape Thunderbolt LTO drive. Reach out for details ✉️

In case you have a huge back catalog of tapes made with a different piece of software, and feeding all those tapes to Canister is just too much work, let us know — we're happy to help with a conversion tool to Catalog your complete archive.