Australian post-production facilities chop render hours on Marvel-based PlayStation series ‘Powers’ with PipelineFX’s Qube!
Bridging the Australian VFX and post-production studio’s Sydney and Brisbane facilities, Cutting Edge has made a recent investment in Qube!, PipelineFX’s render farm management tool tailored to meet the needs of large-scale VFX projects.
Since 1992, Cutting Edge has built a string of studios that employs over 150 artists. Like most professional shops they can do it all: films (Predestination, The Age of Adaline); commercials, and TV series, including Powers, PlayStation Network’s first scripted show based off of the beloved graphic novel series from Marvel.
Plans to link the render farms in Brisbane and Sydney started with a test. The idea was to put render farm managers against each other in a multi-site battle, utilizing both live and offsite workers. This became the proving ground for key needs like: global log files tracking, scripting, platform compatibility and project mirroring. After tests totaling 10,000 jobs, Qube! was declared the winner, in large part, because of how well it could handle the logs.
“With other render management tools, reading log files for machines that aren't local was problematic, and reading log files is an incredibly important part of seeing what's going on and troubleshooting,” said Cutting Edge VFX supervisor Rangi Sutton. “Our TDs and support crew can now dial up logs for a given job through GUIs or the command line tools supplied on any workstation. It’s the only way to see how everything's going across the sites.”
By running off a single dispatcher, artists can submit jobs onsite or remotely through VPN and get access to the power of two farms. With this single point of control, Cutting Edge can now count the software licenses for applications such as The Foundry’s NUKE for all users over the group. Previously, license allocations needed to be predefined between Sydney and Brisbane. During the day the farms consist of dedicated slave units, but after 7:00PM all idle workstations join the pool, offering a 20-30 percent boost in rendering power every night.
Another key feature that drew the team was Qube!’s native compatibility with OSX and Linux, and the Unix-like design. “We already have in-house tools to manage our environment for our major applications: [Autodesk] Maya, [MAXON] C4D, NUKE, [Side Effects Software’s] Houdini, so we can control the specifics of the software used per project and even per shot. While Qube! provides a lot of built-in job types that would help smaller studios kick things off, they don’t get in our way when we choose to ignore them. Essentially, the way we initialize a session at the workstation on the terminal, is exactly the same as what happens on the render farm. We don’t run into new sets of problems for farm-rendered jobs such as user permissions or application versions.”
Running simulations in Houdini proved to be another source of time-savings thanks to per-frame dependencies. In the past, the team found their system was dependent on stages completing before the render could move on. With Qube! all that changed.
“Now we can send stages to the Supervisor and tell Qube! how to perform the operations: pre-processing, simulations, post-processing, followed by rendering. We can configure the granularity of these dependencies; simulations will require all the pre-processed geometry to be available before commencing, whereas a Mantra render need only wait for all the geometry for its given frame.
Using the Qube! Python API, we can describe all these stages from inside of Houdini's Python interpreter, and submit the whole dependency tree in one go,” said Sutton. “Retakes of big effects scenes can be re-simulated and re-rendered through on a single-button click, without any unnecessary idle render nodes or wrangler intervention.”
In the near future, Cutting Edge will also be able to do something that was impossible before: track farm data for every project. “We had access to resource tracking -- how long a crew has been working on individual shots, different storage stats -- but never a stat like comparing CPU cycles against project size. With 50,000 render jobs over the last six months, it’s the aggregate data that Qube! is recording in its data warehouse that will help us make a lot of important strategic decisions. Now we can apply cost-accounting to the render usage, as well as make software license decisions based on the actual usage over time.”