REAL STEEL Process + Design: a case study of
working in virtual production
by Judy Cosgrove
“…[Virtual Production] is a radical
departure from the way we think about film production and the way that we will
think about narrative. It is the beginning of a new conceptual and visual
language."
Jeff Wisniewski, Virtual Art Director, Real Steel,
Tintin, Avatar
I wrote a piece earlier about the success of digital previs as the new hub of efficient
production planning, citing Real Steel, and
how according to the Producers, Previs and Virtual Production kept the film within the parameters of their studio
budget and timeframe.
Real Steel has a seamless blending of real and virtual, and is the
first of its kind to be made in virtual production both in studio and on
location.
PROCESS
Virtual production is the process of shooting
a movie with real-time computer graphics in a collaborative and interactive
way. The CGI can be (but is not limited to) the actor’s mo-cap performances
composited in camera with a CGI environment or real-world environment and vice
versa. This process is used extensively in the creation of modern video games,
and was further developed for filmmaking by James Cameron on Avatar.
In Real Steel, pre-produced
motion-captured virtual performances of digital robot characters were
integrated (through in-camera playback) with real actors performing, and shot
live action, in real time, in the real world. Virtual Production as a non-linear and iterative process was
proved effective beginning with virtual design and set elements (digital
assets) created within the Art Department.
The Production Designer
Led by Production Designer Tom Meyer, the Art Department created not only the
environments but also the hero robot characters for Real Steel; unique given that
animated characters are usually designed by animators, VFX artists, or
sculptors in “creature shops.”
DreamWorks wanted to keep the robot designs tied closely to the overall
concept of the film, and were looking for a candidate to lead both the environment
and character design.
Pre-Production
Meyer started preproduction on September 22, 2009, heading up the team
of concept artists and began collaboration on the hero robots, taking the lead
in designing the robot character of Atom himself. Initial concept artwork
illustrated in 2D was then
modeled in 3D or directly modeled in 3D from the start. Meyer prefers to
finalize designs in 3D rather than rely solely on traditional 2D illustration,
since 3D assets can be used by VFX vendors and without translation. Legacy
Effects, the company that created animatronic versions of the robots worked directly
from these digital files, and as did Digital Domain for the CGI versions.
Designing in 3D, with the aid of computer animation, allowed Meyer to
develop the robot characters more fully through the exploration of their
movement. He could show how hero characters Atom and Zeus might interact with
each other under realistic lighting conditions and suggest camera angles.
Texture, mood and attitude could be conveyed more concretely than in 2D, and by
the first week of October he was quickly sharing these ideas with the director
and the production team.)
These same concepts applied to environments for doing virtual production.
Working in a virtual paradigm first created in the making of Avatar, it was clear that digitally
designed set “assets” were needed prior to the mo-cap shoot. Executive Producer
Josh McLaglen had asked Meyer for the fight containment (boxing ring) to be
designed by the end of January. Meyer agreed and realized they would need to
see whatever was behind them in the virtual camera as well, so in fact he had
all of the sets they needed virtual filming completed by the end of February.
The robot designs were well under way by the second week of October when
Meyer started scouting Detroit. The set
designers began to work from LA off architectural plans with minimal dimensions
and reference photographs. The set designs were developed virtually in 3D, sometimes
independent of their Detroit environs, which were to be determined later, such
as arena architecture that was comped and textured in post.
Having an
accurate model of design for the location in the computer allowed time for
creative experimentation for shots ahead of time, creating a template for the
shooting location. One example is a field
scouted outside of Detroit. Meyer had a
virtual model created of the fairgrounds design for that field (could be any field) in Rhino and animated
the opening shot of truck’s approach to the fairgrounds in Maya within the Art
Department. All other 3D
set designs created in the Art Department were all looked at in previs in
conjunction with VFX Supervisor Erik
Nash (Digital Domain) to figure out what really needed to be built and what
would be created as digital set extensions. . The Art Department provided cross-platform .OBJ
files for Previs Supervisor Casey Schatz
(Giant) to break down for previs, and to work with Meyer exploring the
design in the location virtually with time of day, light and color.
(Interesting
to note that the previs team from Giant Studios did their own 3D lidar (laser)
scans to generate accurate models of the environments (built and existing) once
on location to aid VFX and post production as per usual. Why not give to art
department up front? Industry standards for Virtual Production are being
addressed. This is discussed further in my concluding remarks)
Supervising Art Director
Seth Reed explains: “There were no Art Department scouts to the locations in Detroit, no
site surveys prepared until we got there as an entire art department at the end
of March. A certain amount of work was done while in LA, but it was only work
that we thought we could use in open spaces where we were building from
scratch. Even then, making
assumptions about what could be done was tricky.”
“As Supervising Art Director”
says Reed, “I still did the same thing I always do, in managing the Art Department
- drawings, models, illustrations, budget, schedule, coordination with the
producers, graphics, construction, but it also includes translating into the
physical world, the work of the Virtual Art Department’s mo-cap work. It's
great to design virtually, to model and to previs. Often, these things
still need to be built though. At that point, we are back to practical
matters - how would this really stand up or how would this really be built?
On this shoot there was a tremendous overlap with mo-cap and the virtual
art departments 3D modeled designs. There was a
whole new area to figure out - what had already been shot? What were our
limitations when designing the physically built set that would actually be shot
live? We had to find locations that fit and build practical sets that were
interactive with virtual characters. When a robot was hit and reached
back to touch a rope or a rock or another object, this object had been shot
already as a piece of foam - we had to supply the object touched, in the exact
position and placement already shot, per dimensions and scale adjustments
provided by VFX. The in-camera work that you refer to is only a partial.
Much of our work had already taken place by the time the company got
there to shoot.”
(Reed has some recommendations for ‘best practices’ that I’ve included
at the end of this article.)
Jeff Wisniewski, Art Director Virtual Art
Department and Mo-cap worked within his
department to “strip down” high-poly models created in the Art Department and
hand off to Giant Studios, the company doing the mo-cap shoot. ‘Strip down’ refers
to reducing the poly-count (number of facets) in order to function in Motion Builder,
the primary software tool used for real-time rendering when doing virtual
cinematography. Wisniewski states “I always have one person on my team
designated to stripping models for that reason.”
Previsualization
“What used to be a very obtuse
experience is now very direct and realistic in terms accountability and budgeting”,
says Meyer. “All departments collaborate and decide on what is necessary and
affordable and the director gets what he sees in the concept phase when he
arrives on location. You are not selling
an abstract idea. In addition, from a budget standpoint they spent a fraction
of what they might have spent in the past, before having this technology. This tool effectively collapsed a difficult
design process, turning it into something that could be easily understood.”
Previs Supervisor Casey Schatz emphasizes the
need in previs to respect the physics and limits of the live-action shooting
environment. He acts as a “reality cop” and avoids letting the filmmaker be
seduced by the possibilities the computer is capable of, or any moves that do
not obey the live-action world. On Real
Steel, Schatz programmed safeguard markers into the virtual camera for that
purpose. Indicators would flash on the monitor whenever a move made virtually
would not be possible on site at the location.
Schatz textured
and lit set design models under the direction of the Art Department. He
combined models of the virtual sets with Giant’s models of the locations
generated from their surveys and lidar scans along with his own virtual proxies
of the camera equipment being used. In this way, Schatz could be certain of
real-world parameters. His goal was to give the director an experience as close
as possible to being on location in Detroit.
For the real
world build in the Metal Valley sequence, previs was used to explore the way
the same cliff (built set) could be shot to make Meyer’s cliff design appear
twice as high, by set-ups one way as top on the cliff, then the reverse for the
bottom of the cliff. In addition, this
particular set-up on location required a number of large cranes for several
different departments. Schatz was able
to trouble-shoot the crane placement in the limited space virtually.
Previs proved
to be an invaluable asset again when the location changed at the last minute
for the Zoo sequence. The in-camera fight work had be done in advance, so could
that be adapted anywhere, but at the new location Schatz noticed there were
obtrusive shadows cast from existing structures. He was able to reproduce the
location virtually and with additional programming, correctly project the sun
with time of day, latitude and longitude, for and accurate light study
predicting the fall of the shadows in advance.
Motion-Capture
Shoot
Art
Director Jeff Wisniewski supervised the 6-week motion-capture portion of
Real Steel that began early February 2010 on stage at Giant Studios. He was already an expert in the process,
having been Art Director on Avatar and Tintin, two productions using
motion-capture, virtual camera, and simulcam. The mo-cap shoot now in virtual
production was not for previs, but real footage that would be played back in
simulcam during the live action shoot on location.
Wisniewski oversaw mapping the
pre-designed virtual environment to the stage and the providing of proxy set
pieces in full scale for the actor to interact with as needed. This included
creating construction drawings for any props and set pieces that had motion
(such as a gate) or an actor otherwise interacted with. (They
built a mock-up for the animated bull that they rolled around as part of the action for the bullfight in the
first part of the film. The motion capture from this mock-up gave VFX enough
information to create an incredibly realistic finished product.
During this part of the filmmaking
process the actors performed choreographed fight sequences in the full-scale
proxy environment. Mo-cap cameras rigged around the volume space tracked and
recorded their movement. This information was fed into a computer and the actor’s
performance was targeted onto the pre-loaded digital robot characters. Using
the virtual camera tool, the Director shot video-game resolution sequences of robots
fighting in real-time within the digital environments generated by the art
department. Motion-capture
for the robot fight sequences was completed by March 14, and Wisniewski went on
to prep the location.
Once in Detroit, Wisniewski oversaw the
construction of the main boxing rings at Cobo. He noted how by working
virtually they were able to solve the scale-offset problem created by the 8’
robots, but the real challenge was in physically constructing them on location.
“The boxing ring at Giant Studios was scaled for human actors. The boxing rings
in Detroit, where they shot the giant robots using simulcam needed to be scaled
up 125% in the real world, (a 6’ tall person on
stage had to be an 8’ tall robot at practical location) which is huge.” He
collaborated extensively with each department to bring both the digital sets on
physical sets on budget. The digital files also enabled Wisniewski to contract with
local car parts manufacturers using CNC to fabricate the over-scale parts needed
to build the ring.
DESIGN
The Story World
Real Steel is based on “Steel”, a short
story by Richard Matheson published in 1956 and later made into an episode of
the Twilight Zone in 1964. Screenwriter John Gatins adapted the original story
and shifted the script to focus on the relationship between the father and son
in more of an “Americana” setting. The story
takes in place in the near future, 2020, and revolves around a down-on-his-luck ex-boxer and his young son struggling to make their way
in the new world of robot boxing as fighting ‘bot owners.
The world of the future in Real Steel
closely resembles the world of today, with some advanced technology such as
cell phones and computers. Meyer worked
closely with DreamWorks product placement partners HP, NOKIA, and BEATS to
develop and design new “concept” products that hewed to the ideals that the
companies were developing for their own future lines. Cadillac provided a concept
car driven by Charlie Kenton. (need pic)
Detroit, a classic American city grounded
in manufacturing, was an ideal location for the movie according to Meyer who
was interviewed by Brian Gallagher (MovieWeb.com) while on location:
Can
you talk about filming here in Detroit and how that has influenced your design?
Tom
Meyer: “Michigan
in general has a lot of great things…We filmed in the original Model-T factory
in Highland Park, which we re-purposed as an industrial opera house...There are
also beautiful rolling hillsides …and these classic small towns, with the
courthouse in the center. You get that whole mesh of Americana, this cross-section
of history and technology.”
Can
you talk about your inspirations for the WRB?
Tom
Meyer: “. The
physicality of boxing is about energy and energy displacement, at its core.
It's two guys beating each other up, but when you think about what the ring is
designed to do, you have a sprung floor that bounces with the boxers, you have
ropes that absorb, it's all about energy absorption. …with huge steel cables
with lots of spring and tension and all this cable coming off is just an
extension of the corner post. It's kind of like a turnbuckle on steroids. It's
to help see that energy, so [the robots] feel somewhat humanized… Every single
nut and bolt was custom fabricated. 80% of our crew is from the industrial base
of Detroit, the unions, steelworkers, and carpenters. It's a manufacturing
town.”
The
Robot Characters
The main challenge in designing the robot characters according to Meyer
was in avoiding all established franchises. The robots needed to have a utilitarian
look to them and look able to function in reality, complete with gears and
pistons, and also retain an emotional humanistic quality. The decision was made
early on to have Legacy Effects create full-scale practical animatronic
versions the hero robots Atom, Ambush, Noisy Boy, Axelrod (robots that Kenton
owns and/or works on), in addition to the digital versions created by Digital
Domain.
Concept Artists within the Art Department worked in close collaboration
with Meyer in giving each robot a distinct personality. Tim Flattery, the chair of Art Center’s Entertainment Design
program designed Spitfire, Albino, Axelrod, Twin Cities and Midas. “These aren’t your
typical robots,” said Flattery of the characters he and other concept artists designed
(including fellow Art Center faculty member Daren
Dochterman, Simon Murton, Andrew Leung and Victor Martinez.). “They’re all very stylized and ridiculous, yet
somehow, in the world of the movie they make sense.”
Digital Domain worked with Art
Department and Legacy to finalize robot designs and mechanics that could be
applied to practical robots and CG models alike. “The practical Legacy ‘bots
were invaluable for lighting and texture data,” according to VFX Supervisor Erik Nash, “as they
provided a tangible point of [lighting] reference for digital characters that
needed to be indistinguishable from the real thing. Digital Domain used the Art
Department 3D digital assets directly to model, texture, and rig eight unique,
hero robots for the fight sequences: Ambush, Noisy Boy, Midas, Atom, Metro,
Blacktop, Twin Cities, and Zeus, in addition to numerous background robots that
appear throughout the film… “(Artofvfx.com)
Legacy Effects built three
practical robots – Ambush, Noisy Boy and Atom plus partially destroyed Axelrod
– that were used extensively throughout production for shots requiring human
contact and upper body animation. They started
by printing 3D prototypes at 1/5 scale from the Maya files that originated in
the Art Department. They
exploded the files and broke down into all the parts to be assembled. In this
way they were able to fabricate multiples of separate parts.
CONCLUSION
Real Steel was
a perfect situation for virtual production but this film could not be made
without the technical prowess of every department involved, and I would be
remiss not to mention Virtual Production
Supervisor Glenn Derry, who developed virtual camera and simulcam systems
originally for Avatar. His method of tracking virtual camera data to feed Avid
editorial
is a tremendous
aid to effective collaboration. This method generated useful information for
the previs team and helped the Assistant Directors plan their days by enabling
them to sort shots and group like shots together. Derry’s mobile “live action engineering”
truck allowed them to edit sequences in Avid on site and to be sent back to
Digital Domain who could turn around final render quality shots within a day
during principal photography.
Virtual production allows for
real-time creative collaboration between all departments during production. For
Virtual Production to be effective, it requires early commitment to design and
directorial decisions. When you are in Virtual Production, you are not
previsualizing; you are making the movie.
Wisniewski: "Because the
technology works so well I am not sure we understand the accomplishments made
on Real Steel. This film is a hybrid form that will influence virtual filmmaking.
It is a platform to blend the future, the past, the present and the imagination
seamlessly, it is hyperviz.”
THE LEARNING CURVE
Virtual Production is a
“front-loaded process” that moves away from the traditional mindset of
“figuring it out in post” where many problems are solved after the fact. Design
and previs is key to early collaboration for all departments including
cinematography.
Real Steel Producer Steve Molen (DreamWorks)
spoke from the audience at a Real Steel panel presentation I attended for the
Virtual Production Committee*: “There are no lines
between preproduction and post- it’s all one process.” He urged that Previs/VFX
companies “sweep proprietary software aside” and use “tools involved on the
virtual side that are ingestible and compatible throughout the process. It
would be great if everyone used the same bridging tools. Too many VFX and
previs companies have proprietary software that they end up using as shadow
programs… We should be pushing the [virtual production] process forward in the
same direction.”
*The Joint Technology
Subcommittee on Virtual Production Committee is a round-table of industry
professionals VPC) co-chaired by David
Morin and John Scheele, is currently
engaged in steering the industry toward ‘best practices.’
Supervising Art Director Seth Reed
looks back on his experience on Real Steel and offers some suggestions for the
Art Departments working Virtual Production going forward:
1. I would love to learn more about Lidar,
especially the latest. The information that we had on RS was useful but
was not accurate enough to use for more than conceptual work.
2. Though many parts of the virtual movie were
shot prior to arrival in Detroit, there was still much remaining and we were in
the curious position of finding locations and designing sets for sequences that
had already been shot.
3. Having the virtual Art Director, Jeff, with us
on location was essential. He participated in all that had occurred on
stage plus he had a fantastic communication with the VFX people. He was the
key, lynchpin between Art and VFX.
4. We
are still looking for a more streamlining between Maya (or Rhino, etc) and
Motion Builder. Everything must be converted, this can mean drawing
things twice, especially if the Maya models become very complex.
5. We learned a lot on this, it was a process.
Both worlds affected each other, the practical and virtual. We made
a virtual reality at Giant; we then had a physical reality on location that the
virtual robot actually interacts with. We had to build that
reality! Example, we captured the Zoo sequence on stage, in a ring, with
ropes. Then, we found a great location, an abandoned zoo. We re-contexted
our ring footage to the practical location that they are fighting around rocks
and fallen trees and the robot falls against a rock instead of a rope!
This was a great blend of VFX and Art Departments working together.
Art
Department Credits
Tom Meyer –production designer
Seth Reed – supervising art director
Jeff Wisniewski - mo-cap art director/
locations
Jason Baldwin –Stewart – art director
Tino Schaedler - art director
Bradley Rubin – assistant art director
Trevor Goring- storyboard artist
Joe Venti – storyboard artist
Michael Anthony Jackson – storyboard
artist
Daren Dochterman – concept artist
Tim Flattery – concept artist
Andrew H. Leung – concept artist
Victor J Martinez – concept artist
Simon Murton- concept Illustrator
Jamie Raima – concept illustrator
Miles Teves –concept artist
Michael C. Biddle –set designer
Tim Croshaw – set designer
Tex Kadonaga – set designer
David Moreau – set designer
Anne Porter – set designer
Theodore Sharps set designer
Mike Stassi – set designer
Andrea Carter – art dept. coordinator
Jourdan Henderson – art depr
coordinator
Jennifer Bash – art dept. assistant
Rick Radomski - art dept. assistant
Lisa Fiorito – art dept. researcher
Ceri Glowacki – art dept. production
assistant
William Eliscu – lead graphic designer
Megan Greydanus – graphic designer
Mike Maher – graphic designer
