When the storage connected to its rendering farm started to feel the strain of rapid data growth, animation studio Rainmaker Entertainment improved performance and extended the life of its scale-out NAS
Rainmaker handles animation and rendering for four to six projects a year, ranging from feature movies to DVDs for children and digital games. It has 375 artists working a render farm -- a high-performance computer cluster used for animation -- running applications such as AutoDesk Maya and The Foundry’s Nuke at its studio in Vancouver, British Columbia.
Ron Stinson, Rainmaker’s director of IT and operations, said Rainmaker’s storage grew from 200 1U servers and 18 TB of usable storage six years ago to 700 servers and 320 TB of usable storage today. Most of the storage is on Hitachi Data Systems BlueArc Titan NAS filers, with about 100 TB of secondary storage on EMC Isilon IQ3000 clusters.
But that rapid growth began taking its toll, and Stinson said he started searching for a solid-state caching appliance that could sit in front of his storage. He said his dual-head Titan 3210 was holding up well but he started to notice a slowdown during peak times such as the end of projects or when projects overlapped.
“I knew we were going to saturate the heads on our systems,” he said. “When everybody’s hitting the render farm and pushing files back and forth, that’s when we start to see what users refer to as a network slowdown. Instead of taking 20 seconds to load Nuke, it takes a minute or two. Or loading a scene takes five minutes instead of one minute. That’s when I started talking to all vendors saying ‘We need a solution.’”
That was more than a year ago and the product he ended up with didn’t exist at the time, so Stinson started calculating cost for adding a third Titan head. “That would buy us a little breathing room, but we’d still hit that bottleneck eventually,” he said. “Then what? We buy a fourth head. With a fourth head, we’re pushing the throughput problem off the head to the disk on the back end because the disk is not going to be able to keep up.”
Late last summer, Rainmaker’s VAR pointed out Alacritech’s ANX 1500 appliance. The vendor sent an appliance to Rainmaker to test in late November with 2 TB of solid-state drives (SSD) and 48 GB of DRAM for caching, and two 500 GB hard drives. Stinson said he didn’t have a lot of time to make his decision, because Rainmaker had three projects scheduled for delivery in April or May 2012.
Rainmaker tested the Alacritech caching device on a 16-server chassis for a week, and then put the entire render farm behind it just before Christmas. “We said it’s either going to work or crash and burn,” Stinson said. “It stood up to the entire [render] farm. We found the load on BlueArc dropped from 70 percent to 100 percent utilization to an average of 40 percent to 60 percent. That meant renders were going through quicker.”
Gearing up for multiple projects, Rainmaker ordered more Alacritech appliances. It currently runs them in two pairs for failover. Stinson said his users appreciate that the ANX150 devices help load scenes and applications faster.
“Taking the load off our BlueArc filers has helped in other ways too,” he said. “One is backup. I’m turning over 2 TB [per] night in changes. Our backup library and server have dedicated priority to BlueArc now. They’re not fighting with the render farm to get files up, so our backups are faster.
“It’s giving our storage platform a longer life,” Stinson said of Alacritech’s caching. “Maybe I’ll add another disk for capacity, but I don’t have to seriously look at replacing my heads until a year or two from now.”
The ANX 1500 is a first-generation product, and Stinson said he would like to see more granular reporting -- a request he said he makes to all vendors. “I would like to know how much per file type it’s caching for me so I can fine-tune where I’m storing different files on my storage. For example, if [Apple QuickTime video files] are being hit the most, I can move them from SAS drives to Fibre Channel drives on the BlueArc,” he said.
Another feature he’d like to see is persistent caching, so “if one file type keeps getting hit, let’s tune it so that file type will stay longer on Alacritech than the other file types.”