File Streaming? Could be Suite

note

The following was written in 2024 some info may be outdated. In particular, Resilio Active Everywhere has since added their own file streaming solutions. I still think Suite works best for small to medium sized studios, however on larger projects and particularly with hybrid workflows, modern Resilio is a great option.

This is an overview of how remote VFX workflows can be improved by adopting Suite Studios.

Introduction

At the core of every VFX workflow is thoughtful consideration of who the users are. Typically this includes artists, TDs, coordinators, supervisors and producers. A good remote workflow, that does right by these users, needs the following qualities...

• Consistency — repeatable workflows and continuity.
• Security — follows best practices - in this case, ideally MPA TPN assessed.
• Efficiency — there’s no place for extensive manual copying and sorting.
• Performance — speedy, snappy, not interfere with work.
• Reliability — reduce bugs and glitches!

To outline a typical VFX pipeline, I'll start with the trusty workstation:

The modern VFX world has mostly gone with ShotGrid’s Pipeline Toolkit as the foundation for many pipeline setups. The user-friendly integrations with their production management software, and support for most digital content creation tools, make this toolkit an essential starting point. Using the toolkit, companies can build things like ingest tools, export tools, environment specific tools, plugin managers and plenty other creative integrations.

One of the basic requirements of ShotGrid’s Pipeline Toolkit is a storage location to work off of. The toolkit looks for a specified file path to read and write files based on the operating system, e.g Windows: P:\YOUR_DRIVE, MAC: /Volumes/YOUR_DRIVE, Linux: /mnt/YOUR_DRIVE.

Let's call our shared storage location "BLOCKY".

The Problem with Remote Work

Many studios are multi-location companies with employees spread out across the country. For this example, BLOCKY is in Los Angeles, but let's say ther artists and other users of our setup are in New York. If I’m working in New York, I can’t specify BLOCKY as the toolkit storage location because I’m not connected to the network BLOCKY is on (unless you run a VPN, which generally don't have strong enough performance).

1. Remote Desktop

The first step in addressing the problem of getting people connected to BLOCKY from afar, is to move people closer to BLOCKY, physically — duh! But if we can't do that, remoting in using an application like Teradici or DCV lets someone in New York or elsewhere connect to a workstation on-premise in Los Angeles that is a part of the network that includes BLOCKY.

A quick comparison of Remote Desktop solutions:

	Jump Desktop	NICE DCV	Parsec	Teradici
Integration with AWS	❌	✅	❌	❌
Price	16/user/month	13/desktop/month	30/user/month	20/user/month
Good Performance	✅	✅	❌	✅
Linux Support	Sort of	✅	In Progress	✅

This is a fast, efficient, and secure way of working. But what happens when you run out of workstations in Los Angeles? Or I have good reasons to want to work locally but in another location?

2. Symlinks & Syncing

Symlinks are your computers way of creating a virtual filepath that matches ShotGrid's specified storage location file path while actually pointing to another location on a local machine. So for example, while the shared storage available in New York might be /mnt/BLOCKY2/, a symlink lets us make that storage also accessible locally from /mnt/BLOCKY/. Great! But with multiple symlinks, and essentially multiple BLOCKIES around the country, I need to sync them so that assets saved to one location can be accessed by ShotGrid toolkit from other locations.

Resilio, Syncthing, and Signiant Jet are all sync solutions that scan for changes in one location and transfer files back and forth to keep multiple servers in sync. This part is pretty straightforward but there are a few things I need to consider: speed, reliability, and integration with the cloud. Starting with Syncthing, it’s supposed to be fast, it has transfer rules that make it quite reliable, but it doesn’t sync with object storage in the cloud. However it is open source and free, so that’s a great plus. Each sync solution comes with its pros and cons. For now I want to try Syncthing, but I neeeded a cloud workflow for when things get busy and I don’t have enough hardware to support the resource demands — is there another way to ‘sync’ these servers?

Here’s a brief comparison of different sync solutions:

	Speed	Price	Cloud Compatible	As Fast as File System
Syncthing	SLOW	Free	❌	❌
Signiant Jet	FAST	Expensive	✅	❌
Resilio	FAST	Mid Range	✅	❌

3. File Streaming

Locally Accessible and Mountable Cloud Storage - maybe making an on-prem server the place you direct ShotGrid to in the first place is a mistake. Instead, one option to consider is a locally mounted cloud bucket. This cloud bucket would be the single source of truth that caches to local storage in these different geographic and cloud places. Enter Lucidlink and Suite.

In this diagram, you can see Suite sits at the center. It’s sitting in the middle because in this scenario, ShotGrid is directed to Y:\Suite on windows or /mnt/Suite/ on linux as its primary storage location.

Here’s a brief comparison of Suite and Lucidlink — just the dealbreakers:

	Suite	Lucidlink
Price	75/TB + 10/user	80/TB + 10/user
Responsive Team	✅	❌
Easy install	✅	❌
Proven Track Record	❌	✅
Faster Speed	✅	❌
Transactional Writes	✅	❌

Aside from replacing a sync solution with cloud integration, Suite has another important place in this workflow. On the Supervisor and Producer side of things, Suite provides access to hi-res movie and EXR versions of shots that can be easily opened and reviewed via RV or SG's interfaces. Anything in the pipeline will be viewable and editable remotely on a laptop by coordinators, producers and supervisors.

note

Media is directly loadable from the “path to frames” and “path to movie” fields that direct RV to EXRs and hi-res MOVs respectively.

Final Thoughts

Suite is a great choice for smaller shops looking to put their workflows online in way that enables great remote work options. With a recent approval from the Trusted Partner Network, Suite is a safe, reliable way to access files on the go from anywhere.

Workflow Before

Workflow After