26 Mar 2020

Rooster Teeth

Rooster Teeth produces the globally acclaimed animated series RWBY ("Ruby"), one of the first western anime series to be distrubuted in Japan; the award winning Rooster Teeth Podcast; Immersion, a reality format that brings video game theory to the real world and several films for Youtube Red.

Rooster Teeth has a massive global footprint of more than 45 million subscribers to its YouTube Network, 5 million unique monthly visitors to its Rooster Teeth.com hub and 3 million registered community members. The company was founded in 2003, and is a subsidary of Otter Media, a Warner-Media Company.

And it all runs on GB Labs SPACE intelligent storage system.

Serveral years ago, Rooster Teeth was invited to take a studio tour of the Austin, Texas office of Megalomedia to have a look at their operations. Megalomedia is a full-service television and film studio that develops and produces orginal content for U.S and international broadcasters and has offices in Austin and London.

Rooster Teeth Director of Technology, Adam Baird, later met with Adi Antariksa - Chief Business Officer (USA) from GB Labs and NAB to talk through the technology he had seen and put it to the test by sitting down and connecting a laptop via 10Gb card. He was soon pulling 1100 MB/s off the available network.

"That's as impressive now as it was then," said Baird. Baird was doing research and testing because Rooster Teeth was growing, rapidly, and needed more performance. At the time, Rooster Teeth was using a legacy Apple X-Serve in a 24 TB configuration that, at the time, was an impressive amount of storage and seemed to perform well.

Image

However, as Rooster Teeth continued to grow, problems began to mount.

According to Baird: "We realised that our storage needed to grow out of being a relatively small home-built system into a full-blown, real world and professionally supported server system that had a mature, vibrant ecosystem surrounding it, which we had found in GB Labs' SPACE."

SPACE is the fastest shared storage on the market and enables its users to work and collaborate on HD and 4K projects, live, accross the network.

By redefining network attached storage (NAS) SPACE is the true successor to fibre channel and iSCSI shared storage topologies, which are notoriously complex to install; difficult to maintain or scale; and typically demand lots of support. SPACE negates all of these issues while delivering unprecendented levels of performance and scalability.

Rooster Teeth needed that power and flexibility because it was operating in a world that could suddenly require 50TB of additional storage server capacity and the only way to acheive that at the time was to format and add more physical drives to a RAID enclosure, an always painful and occasionally terrifying process.

Baird added, "The ability to buy a GB Labs SPACE storage system; have all the drives show up; rack-mount them in 20 minutes; plug in a SAS connector; get on the phone with GB Labs to push the right buttons and suddenly have an additional 80 terabytes of storage was magical to the point of almost not seeming real. It all just worked. It was a revelation.

"Since then, we've done the same thing four or five times and what seemed surreal when we installed the first SPACE has become the norm. In less than an hour we can go from 90 percent of our storage being occupied to being in a much more comfortable spot"

The GB Labs system is used for all of the archiving; editing and general business storage for several of its key departments, the primary one being "Let's Play", a group producers that make video game content for a range of YouTube channels. SPACE is also used by Rooster Teeth broadcast department, which creates approximatley 10-15 terabytes of content each week, 8-15 hours of which results in unique content that goes up on YouTube and to Rooster Teeth's own SVOD,RoosterTeeth.Com

In short, GB Labs storage sits at the heart of a huge amount of all Rooster Teeth broadcast content, putting Rooster Teeth in the same league as very large media companies. Said Baird, "SPACE effortlessly handles an astonishing amount of data in and out every single day."

Rooster Teeth employs approximatley 450 people and, of that, close to 400 use the GB Labs system daily. The core system currently sits somewhere between 550 and 600 terabytes of storage and shared servers that are used by the entire company. In addition to being the primary storage cluster for all broadcast and Let's Play groups, it is also used for all marketing, sales, and operations.

Moreover, it supports 45-50 full-time editors, the majority of whom are in Austin, plus a small group based in Los Angeles, who edit locally and trasnfer content back to Austin. Most editors, color, and VFX operators are 10 Gb connected to Rooster Teeth's back fabric and are able to utilize 100 percent of the system.

Baird said, "All departments making daily content use SPACE. We have two primary SPACE clusters, one of approximately 150 terabytes and another 460 terabyte cluster for daily replications; backup restoration; archiving; etc. We got a nice email notification every night to tell us it has finished.

Rooster Teeth is responsible for 20+ YouTube channels; and the GB Labs storage system also powers all of its shorts that go to Instagram, Twitter, Facebook, etc... so when you look at the real capability for reach at Rooster Teeth, daily viewership is easily 250k+ and often expands to many millions.

Baird said, "All told, we're looking at around 400 to 450 million monthly impressions, and all of the content that emanates from Rooster Teeth, which includes everything that goes on YouTube, Instagram, Twitter, Facebook, our distrubution partners, everything, either goes through SPACE or has lived on SPACE for the entirety of its creation.

Rooster Teeth trailled other storage systems through the years but, according to Baird, "Every single time it's been a really fustrating experience for all of our live divisions". "Our workstations in Texas are designed to dump to Network Storage at a full 10 GB/s, but with other systems we completely bottleneck the NAS before even dumping the footage. This is especially difficult whenever we're in the feild - which is a lot - trying to do DIT file dumps using other systems. Compared to what we're userd to with, SPACE, any other approach is painful, so we don't wanr to be without SPACE."

Supplier and manufactured support is crucial, but there are those who talk it, and those who walk it. Baird said, " We've tried some similar over-the-counter, cheap, easy-to-use plug-in-your-own drives' and on a regular bases we would have a problem that ended up becoming a significant issue that require us to rethink what we're doing and repair things all the way down to rewriting to OS".

"On occasion we have pushed the GB Labs system beyond what it was designed for and it never even whimpered. Still, it was a comfort to know that we had GB Labs experts monitoring the system and working with us the entire time. I can't speak highly enough of the support we've had."

Image

Well, what SPACE has done for our workflows is a little bit of a blessing, and a little bit of a curse.

All of our broadcast divisions used to make a podcast; livestream it; and within 48-hours post the podcast to the internet for people to watch.

It's now reached a point where what used to take 48-hours can be turned around in three to four hours. And that's film, dump to network, index and work with footage, edit, and upload to the internet. The amount of performance we've been able to get out of the system is great, but we've created an expectation within our community that we can release videos three to five hours after they're filmed - every time - which is not always possible!

Because of SPACE, we've become a victim of our own success.

Adam Baird

Director of Technology