on Thursday, March 1, 2012
UPDATE REAL STEEL article with photos in latest issue of Perspective Magazine

REAL STEEL Process + Design: a case study of working in virtual production 
by Judy Cosgrove
“…[Virtual Production] is a radical departure from the way we think about film production and the way that we will think about narrative. It is the beginning of a new conceptual and visual language."
                            Jeff Wisniewski, Virtual Art Director, Real Steel, Tintin, Avatar

I wrote a piece earlier about the success of digital previs as the new hub of efficient production planning, citing Real Steel, and how according to the Producers, Previs and Virtual Production kept the film within the parameters of their studio budget and timeframe. 


Real Steel has a seamless blending of real and virtual, and is the first of its kind to be made in virtual production both in studio and on location.


PROCESS

Virtual production is the process of shooting a movie with real-time computer graphics in a collaborative and interactive way. The CGI can be (but is not limited to) the actor’s mo-cap performances composited in camera with a CGI environment or real-world environment and vice versa. This process is used extensively in the creation of modern video games, and was further developed for filmmaking by James Cameron on Avatar.

In Real Steel, pre-produced motion-captured virtual performances of digital robot characters were integrated (through in-camera playback) with real actors performing, and shot live action, in real time, in the real world. Virtual Production as a non-linear and iterative process was proved effective beginning with virtual design and set elements (digital assets) created within the Art Department.

The Production Designer

Led by Production Designer Tom Meyer, the Art Department created not only the environments but also the hero robot characters for Real Steel; unique given that animated characters are usually designed by animators, VFX artists, or sculptors in “creature shops.”
DreamWorks wanted to keep the robot designs tied closely to the overall concept of the film, and were looking for a candidate to lead both the environment and character design.


Pre-Production

Meyer started preproduction on September 22, 2009, heading up the team of concept artists and began collaboration on the hero robots, taking the lead in designing the robot character of Atom himself. Initial concept artwork illustrated in 2D was then modeled in 3D or directly modeled in 3D from the start. Meyer prefers to finalize designs in 3D rather than rely solely on traditional 2D illustration, since 3D assets can be used by VFX vendors and without translation. Legacy Effects, the company that created animatronic versions of the robots worked directly from these digital files, and as did Digital Domain for the CGI versions.

Designing in 3D, with the aid of computer animation, allowed Meyer to develop the robot characters more fully through the exploration of their movement. He could show how hero characters Atom and Zeus might interact with each other under realistic lighting conditions and suggest camera angles. Texture, mood and attitude could be conveyed more concretely than in 2D, and by the first week of October he was quickly sharing these ideas with the director and the production team.)

These same concepts applied to environments for doing virtual production. Working in a virtual paradigm first created in the making of Avatar, it was clear that digitally designed set “assets” were needed prior to the mo-cap shoot. Executive Producer Josh McLaglen had asked Meyer for the fight containment (boxing ring) to be designed by the end of January. Meyer agreed and realized they would need to see whatever was behind them in the virtual camera as well, so in fact he had all of the sets they needed virtual filming completed by the end of February. 

The robot designs were well under way by the second week of October when Meyer started scouting Detroit.  The set designers began to work from LA off architectural plans with minimal dimensions and reference photographs. The set designs were developed virtually in 3D, sometimes independent of their Detroit environs, which were to be determined later, such as arena architecture that was comped and textured in post.

Having an accurate model of design for the location in the computer allowed time for creative experimentation for shots ahead of time, creating a template for the shooting location. One example is a field scouted outside of Detroit.  Meyer had a virtual model created of the fairgrounds design for that field (could be any field) in Rhino and animated the opening shot of truck’s approach to the fairgrounds in Maya within the Art Department.  All other 3D set designs created in the Art Department were all looked at in previs in conjunction with VFX Supervisor Erik Nash (Digital Domain) to figure out what really needed to be built and what would be created as digital set extensions. .  The Art Department provided cross-platform .OBJ files for Previs Supervisor Casey Schatz (Giant) to break down for previs, and to work with Meyer exploring the design in the location virtually with time of day, light and color.

(Interesting to note that the previs team from Giant Studios did their own 3D lidar (laser) scans to generate accurate models of the environments (built and existing) once on location to aid VFX and post production as per usual. Why not give to art department up front? Industry standards for Virtual Production are being addressed. This is discussed further in my concluding remarks)


Supervising Art Director Seth Reed explains: “There were no Art Department scouts to the locations in Detroit, no site surveys prepared until we got there as an entire art department at the end of March. A certain amount of work was done while in LA, but it was only work that we thought we could use in open spaces where we were building from scratch.   Even then, making assumptions about what could be done was tricky.”

“As Supervising Art Director” says Reed, “I still did the same thing I always do, in managing the Art Department - drawings, models, illustrations, budget, schedule, coordination with the producers, graphics, construction, but it also includes translating into the physical world, the work of the Virtual Art Department’s mo-cap work. It's great to design virtually, to model and to previs.  Often, these things still need to be built though.  At that point, we are back to practical matters - how would this really stand up or how would this really be built?  On this shoot there was a tremendous overlap with mo-cap and the virtual art departments 3D modeled designs. There was a whole new area to figure out - what had already been shot?  What were our limitations when designing the physically built set that would actually be shot live? We had to find locations that fit and build practical sets that were interactive with virtual characters.  When a robot was hit and reached back to touch a rope or a rock or another object, this object had been shot already as a piece of foam - we had to supply the object touched, in the exact position and placement already shot, per dimensions and scale adjustments provided by VFX.  The in-camera work that you refer to is only a partial.  Much of our work had already taken place by the time the company got there to shoot.”

(Reed has some recommendations for ‘best practices’ that I’ve included at the end of this article.)

Jeff Wisniewski, Art Director Virtual Art Department and Mo-cap worked within his department to “strip down” high-poly models created in the Art Department and hand off to Giant Studios, the company doing the mo-cap shoot. ‘Strip down’ refers to reducing the poly-count (number of facets) in order to function in Motion Builder, the primary software tool used for real-time rendering when doing virtual cinematography. Wisniewski states “I always have one person on my team designated to stripping models for that reason.”  


Previsualization

 “What used to be a very obtuse experience is now very direct and realistic in terms accountability and budgeting”, says Meyer. “All departments collaborate and decide on what is necessary and affordable and the director gets what he sees in the concept phase when he arrives on location.  You are not selling an abstract idea. In addition, from a budget standpoint they spent a fraction of what they might have spent in the past, before having this technology.  This tool effectively collapsed a difficult design process, turning it into something that could be easily understood.”

Previs Supervisor Casey Schatz emphasizes the need in previs to respect the physics and limits of the live-action shooting environment. He acts as a “reality cop” and avoids letting the filmmaker be seduced by the possibilities the computer is capable of, or any moves that do not obey the live-action world. On Real Steel, Schatz programmed safeguard markers into the virtual camera for that purpose. Indicators would flash on the monitor whenever a move made virtually would not be possible on site at the location.

Schatz textured and lit set design models under the direction of the Art Department. He combined models of the virtual sets with Giant’s models of the locations generated from their surveys and lidar scans along with his own virtual proxies of the camera equipment being used. In this way, Schatz could be certain of real-world parameters. His goal was to give the director an experience as close as possible to being on location in Detroit.

For the real world build in the Metal Valley sequence, previs was used to explore the way the same cliff (built set) could be shot to make Meyer’s cliff design appear twice as high, by set-ups one way as top on the cliff, then the reverse for the bottom of the cliff.  In addition, this particular set-up on location required a number of large cranes for several different departments.  Schatz was able to trouble-shoot the crane placement in the limited space virtually.

Previs proved to be an invaluable asset again when the location changed at the last minute for the Zoo sequence. The in-camera fight work had be done in advance, so could that be adapted anywhere, but at the new location Schatz noticed there were obtrusive shadows cast from existing structures. He was able to reproduce the location virtually and with additional programming, correctly project the sun with time of day, latitude and longitude, for and accurate light study predicting the fall of the shadows in advance.

Motion-Capture Shoot

Art Director Jeff Wisniewski supervised the 6-week motion-capture portion of Real Steel that began early February 2010 on stage at Giant Studios.  He was already an expert in the process, having been Art Director on Avatar and Tintin, two productions using motion-capture, virtual camera, and simulcam. The mo-cap shoot now in virtual production was not for previs, but real footage that would be played back in simulcam during the live action shoot on location.

Wisniewski oversaw mapping the pre-designed virtual environment to the stage and the providing of proxy set pieces in full scale for the actor to interact with as needed. This included creating construction drawings for any props and set pieces that had motion (such as a gate) or an actor otherwise interacted with.  (They built a mock-up for the animated bull that they rolled around as part of the action for the bullfight in the first part of the film. The motion capture from this mock-up gave VFX enough information to create an incredibly realistic finished product.

During this part of the filmmaking process the actors performed choreographed fight sequences in the full-scale proxy environment. Mo-cap cameras rigged around the volume space tracked and recorded their movement. This information was fed into a computer and the actor’s performance was targeted onto the pre-loaded digital robot characters. Using the virtual camera tool, the Director shot video-game resolution sequences of robots fighting in real-time within the digital environments generated by the art department.  Motion-capture for the robot fight sequences was completed by March 14, and Wisniewski went on to prep the location.

Once in Detroit, Wisniewski oversaw the construction of the main boxing rings at Cobo. He noted how by working virtually they were able to solve the scale-offset problem created by the 8’ robots, but the real challenge was in physically constructing them on location. “The boxing ring at Giant Studios was scaled for human actors. The boxing rings in Detroit, where they shot the giant robots using simulcam needed to be scaled up 125% in the real world, (a 6’ tall person on stage had to be an 8’ tall robot at practical location) which is huge.” He collaborated extensively with each department to bring both the digital sets on physical sets on budget. The digital files also enabled Wisniewski to contract with local car parts manufacturers using CNC to fabricate the over-scale parts needed to build the ring. 


DESIGN
The Story World

Real Steel is based on “Steel”, a short story by Richard Matheson published in 1956 and later made into an episode of the Twilight Zone in 1964. Screenwriter John Gatins adapted the original story and shifted the script to focus on the relationship between the father and son in more of an “Americana” setting.  The story takes in place in the near future, 2020, and revolves around a down-on-his-luck ex-boxer and his young son struggling to make their way in the new world of robot boxing as fighting ‘bot owners.

The world of the future in Real Steel closely resembles the world of today, with some advanced technology such as cell phones and computers.  Meyer worked closely with DreamWorks product placement partners HP, NOKIA, and BEATS to develop and design new “concept” products that hewed to the ideals that the companies were developing for their own future lines.  Cadillac provided a concept car driven by Charlie Kenton. (need pic)

Detroit, a classic American city grounded in manufacturing, was an ideal location for the movie according to Meyer who was interviewed by Brian Gallagher (MovieWeb.com) while on location:

Can you talk about filming here in Detroit and how that has influenced your design?

Tom Meyer:  “Michigan in general has a lot of great things…We filmed in the original Model-T factory in Highland Park, which we re-purposed as an industrial opera house...There are also beautiful rolling hillsides …and these classic small towns, with the courthouse in the center. You get that whole mesh of Americana, this cross-section of history and technology.”

Can you talk about your inspirations for the WRB?

Tom Meyer: “. The physicality of boxing is about energy and energy displacement, at its core. It's two guys beating each other up, but when you think about what the ring is designed to do, you have a sprung floor that bounces with the boxers, you have ropes that absorb, it's all about energy absorption. …with huge steel cables with lots of spring and tension and all this cable coming off is just an extension of the corner post. It's kind of like a turnbuckle on steroids. It's to help see that energy, so [the robots] feel somewhat humanized… Every single nut and bolt was custom fabricated. 80% of our crew is from the industrial base of Detroit, the unions, steelworkers, and carpenters. It's a manufacturing town.”


The Robot Characters

The main challenge in designing the robot characters according to Meyer was in avoiding all established franchises. The robots needed to have a utilitarian look to them and look able to function in reality, complete with gears and pistons, and also retain an emotional humanistic quality. The decision was made early on to have Legacy Effects create full-scale practical animatronic versions the hero robots Atom, Ambush, Noisy Boy, Axelrod (robots that Kenton owns and/or works on), in addition to the digital versions created by Digital Domain.

Concept Artists within the Art Department worked in close collaboration with Meyer in giving each robot a distinct personality. Tim Flattery, the chair of Art Center’s Entertainment Design program designed Spitfire, Albino, Axelrod, Twin Cities and Midas. “These aren’t your typical robots,” said Flattery of the characters he and other concept artists designed (including fellow Art Center faculty member Daren Dochterman, Simon Murton, Andrew Leung and Victor Martinez.). “They’re all very stylized and ridiculous, yet somehow, in the world of the movie they make sense.”

Digital Domain worked with Art Department and Legacy to finalize robot designs and mechanics that could be applied to practical robots and CG models alike. “The practical Legacy ‘bots were invaluable for lighting and texture data,” according to VFX Supervisor Erik Nash, “as they provided a tangible point of [lighting] reference for digital characters that needed to be indistinguishable from the real thing. Digital Domain used the Art Department 3D digital assets directly to model, texture, and rig eight unique, hero robots for the fight sequences: Ambush, Noisy Boy, Midas, Atom, Metro, Blacktop, Twin Cities, and Zeus, in addition to numerous background robots that appear throughout the film…(Artofvfx.com)

Legacy Effects built three practical robots – Ambush, Noisy Boy and Atom plus partially destroyed Axelrod – that were used extensively throughout production for shots requiring human contact and upper body animation.  They started by printing 3D prototypes at 1/5 scale from the Maya files that originated in the Art Department.  They exploded the files and broke down into all the parts to be assembled. In this way they were able to fabricate multiples of separate parts.


CONCLUSION

Real Steel was a perfect situation for virtual production but this film could not be made without the technical prowess of every department involved, and I would be remiss not to mention Virtual Production Supervisor Glenn Derry, who developed virtual camera and simulcam systems originally for Avatar. His method of tracking virtual camera data to feed Avid editorial
is a tremendous aid to effective collaboration. This method generated useful information for the previs team and helped the Assistant Directors plan their days by enabling them to sort shots and group like shots together.  Derry’s mobile “live action engineering” truck allowed them to edit sequences in Avid on site and to be sent back to Digital Domain who could turn around final render quality shots within a day during principal photography.

Virtual production allows for real-time creative collaboration between all departments during production. For Virtual Production to be effective, it requires early commitment to design and directorial decisions. When you are in Virtual Production, you are not previsualizing; you are making the movie.

Wisniewski: "Because the technology works so well I am not sure we understand the accomplishments made on Real Steel. This film is a hybrid form that will influence virtual filmmaking. It is a platform to blend the future, the past, the present and the imagination seamlessly, it is hyperviz.” 


THE LEARNING CURVE

Virtual Production is a “front-loaded process” that moves away from the traditional mindset of “figuring it out in post” where many problems are solved after the fact. Design and previs is key to early collaboration for all departments including cinematography.

Real Steel Producer Steve Molen (DreamWorks) spoke from the audience at a Real Steel panel presentation I attended for the Virtual Production Committee*: “There are no lines between preproduction and post- it’s all one process.” He urged that Previs/VFX companies “sweep proprietary software aside” and use “tools involved on the virtual side that are ingestible and compatible throughout the process. It would be great if everyone used the same bridging tools. Too many VFX and previs companies have proprietary software that they end up using as shadow programs… We should be pushing the [virtual production] process forward in the same direction.”

*The Joint Technology Subcommittee on Virtual Production Committee is a round-table of industry professionals VPC) co-chaired by David Morin and John Scheele, is currently engaged in steering the industry toward ‘best practices.

Supervising Art Director Seth Reed looks back on his experience on Real Steel and offers some suggestions for the Art Departments working Virtual Production going forward:

1. I would love to learn more about Lidar, especially the latest.  The information that we had on RS was useful but was not accurate enough to use for more than conceptual work.

2.  Though many parts of the virtual movie were shot prior to arrival in Detroit, there was still much remaining and we were in the curious position of finding locations and designing sets for sequences that had already been shot.

3.  Having the virtual Art Director, Jeff, with us on location was essential.  He participated in all that had occurred on stage plus he had a fantastic communication with the VFX people. He was the key, lynchpin between Art and VFX.

4.  We are still looking for a more streamlining between Maya (or Rhino, etc) and Motion Builder.  Everything must be converted, this can mean drawing things twice, especially if the Maya models become very complex.

5.  We learned a lot on this, it was a process.  Both worlds affected each other, the practical and virtual.  We made a virtual reality at Giant; we then had a physical reality on location that the virtual robot actually interacts with.  We had to build that reality!  Example, we captured the Zoo sequence on stage, in a ring, with ropes.  Then, we found a great location, an abandoned zoo.  We re-contexted our ring footage to the practical location that they are fighting around rocks and fallen trees and the robot falls against a rock instead of a rope!  This was a great blend of VFX and Art Departments working together.

Art Department Credits
Tom Meyer –production designer
Seth Reed – supervising art director
Jeff Wisniewski - mo-cap art director/ locations
Jason Baldwin –Stewart – art director
Tino Schaedler - art director
Bradley Rubin – assistant art director

Trevor Goring- storyboard artist
Joe Venti – storyboard artist
Michael Anthony Jackson – storyboard artist

Daren Dochterman – concept artist
Tim Flattery – concept artist
Andrew H. Leung – concept artist
Victor J Martinez – concept artist
Simon Murton- concept Illustrator
Jamie Raima – concept illustrator
Miles Teves –concept artist

Michael C. Biddle –set designer
Tim Croshaw – set designer
Tex Kadonaga – set designer
David Moreau – set designer
Anne Porter – set designer
Theodore Sharps set designer
Mike Stassi – set designer

Andrea Carter – art dept. coordinator
Jourdan Henderson – art depr coordinator
Jennifer Bash – art dept. assistant
Rick Radomski  - art dept. assistant
Lisa Fiorito – art dept. researcher
Ceri Glowacki – art dept. production assistant

William Eliscu – lead graphic designer
Megan Greydanus – graphic designer
Mike Maher – graphic designer

0 comments:

Post a Comment