Multimedia
Blogs
Forum
Affordable HD
Whitepapers
Newsletters
Events
Advertisers
Blogcast
MillimeterCreating Virtual Performers
Apr 1, 2003 12:00 PM, By Ellen Wolff
Disney’s Human Face Project
 The CG figure on the left was generated from the actual actor on the right using a process called performance capture "cross-mapping." As opposed to conventional motion capture, performance capture focuses more on overall motion, not the motion of specific coordinates. |
Of all the recent developments in computer animation, few hold the promise of Disney's Human Face Project, spearheaded by one of CGI's genuine pioneers, Lance Williams.
Even in a year marked by notable advances in computer vision techniques — which allowed filmmakers to extract 3D information from 2D photography in order to add 3D-CG elements to live action — the approach developed by Williams' team stands out. When the Human Face demo was unveiled at Siggraph 2002, its implications for visual effects, animated films, videogames, and even animatronics couldn't be denied.
Filmed by Visual Effects Oscar-winner Hoyt Yeatman, the demo simply shows two men sitting side by side at a gaming table, pleasantly talking to one another. What's striking is the resemblance between them. Were it not for an obvious difference in their ages, they could be twins. But in fact, the older man is the only real actor. The photo-real 3D-CG face of his youthful clone was brought to life through a process that Lance Williams characterizes as “cross-mapping.” “We tracked the face of a guy in his sixties onto a guy in his thirties. We abstracted the performance of the actor in a way that allowed us to cross-map it to another face.”
This cross-mapping technique, developed by Williams' 12-person team over the past couple of years, was initially prompted by the demands of a film that ultimately was shelved. Yeatman, slated to supervise that film's effects through Disney's FX arm The Secret Lab, explains the challenge. “The impetus was to be able to have a famous actor with a long film history — like a Sean Connery — and have a younger ‘double’ that would show the actor at about 25 years of age. At first they thought, ‘Let's use makeup.’ But making someone look young is not a process of adding stuff — it requires removing stuff. Then they also tried 2D effects, but that looked like a bad burn victim.”
When Yeatman was brought in, he recalls, “I thought we should look at the idea of doing what I would call digital makeup. We would essentially have the mature actor driving the facial performance of his double. There were a number of techniques out there, all of which were really just motion capture. No one had really conquered the idea of capturing performance.”
Conventional mo-cap processes, in which a performer is fitted with reflective markers and then filmed with multiple cameras, really just records a shell, Yeatman says. “And that's the problem. If you go to edit it, or apply it to another face with a different shape, you know nothing of the performance — you just know the physical 3D coordinates.”
In contrast, Williams and his collaborator Walt Hyneman came up with a concept not unlike what musicians do when they use MIDI technology to translate a performance made with one instrument onto another.
Describing how this worked for the Human Face demo, Yeatman says, “We filmed our actor split-screen, sitting at the gaming table. We gathered information on him using just two cameras, almost like a Cinerama setup. One was the production camera. In this case it was a VistaVision camera, which offered us higher resolution. We also had a 4-perf camera off to one side, not even slaved. That's all that we filmed, but it gave us a tremendously rich field of data — as opposed to some poor soul having to glue lots of reflective dots onto someone's face.”
The resulting footage, when scanned into the computer, was subjected to software analysis. While Disney understandably considers the details of this technique extremely proprietary, in essence what it does is track pixels. It looks at the photographed image in both views from the two cameras as each pixel moves from one view to the other. Every pixel making up the face is a target for this optical flow analysis.
In preparation for applying the actor's facial performance to a computer-generated digital double, the Disney team worked with plastic surgeons to identify the roughly 84 muscles that comprise the human face.
Again using a musical analogy, Yeatman likens the movements of different muscle groups to playing chords. “Usually CG animators have sliders that go from happy to sad. But by playing a ‘chord’ essentially, you'd get a group of muscles to move.”
With these sliders in place, the pixel-specific nuances of the actor's facial performance were applied to the digital double's face. The mathematical engine developed by Williams' team looked at those 84 individual controls and tried all the different permutations until it found the right combinations that would drive those 84 sliders to make the young CG face match the facial expressions of the older photographed actor. “Computers are great at comparing,” notes Yeatman.
While this sounds very automated, the results are tweakable, allowing filmmakers to edit them easily by adjusting individual sliders. “Cross-mapping becomes very simple,” says Yeatman, who believes that it's also very malleable. “In this case, we built a younger version of a person. But we also did tests where we mapped human expressions onto an insect. As long as your muscle systems are cross-mapped, just like in MIDI, the data that you've stored is really the performance, not the motion. You can use it to drive the performances of characters with unusual shapes. Eerily enough, you can see the essence of that person in that performance. You could make a worm act like Robin Williams.”
That's a telling analogy, given the great cel animation that was inspired by Robin's expressive face for the genie character he voiced in Disney's Aladdin. While that was obviously done manually, this new technology represents a logical extension of Disney's tradition of photographing reference material for its animators. As Lance Williams observes, “A lot of Disney animation was well supported by shots of actors staging bits of business. What the computer offers in this area is a much more flexible way of acquiring motion.”
In thinking about how the Human Face Project technology might be applied to future animated characters, Disney Technology Vice President John Carey says, “This will allow us even more nuanced performances that we're able to animate by hand. That's the hope. We're trying to develop high-level tools for animation. We believe there's real production benefit to this technology, enabling us to make either more nuanced animation or a different kind of animation. I hesitate to say ‘more real’ because when we capture a performance we may cross it with something that couldn't exist in nature.”
 The captured performance is scanned and analyzed by software to make sure pixels are tracked correctly. |
The prospect of outrageous aliens giving performances driven by great actors is enticing. Yeatman says he can imagine an actor like Christopher Walken creating a creepy creature without undergoing six hours of makeup every day. “You can apply his performance to a CG character that you can squash and stretch in unusual ways. In addition to expanding the abilities of performers, you can generate characters quickly and efficiently,” he says.
Another advantage for visual effects filmmakers is that, unlike capture technologies requiring special stages or infra-red equipment, Disney's approach can be brought on location, and actors can be filmed under regular stage lights or even sunlight. It's been tested successfully using HD cameras as well.
Carey cautions, however, that Disney has tried not to make claims of efficiency. “The driving force behind this is having new capabilities,” he says.
For his part, Williams addresses the question of efficiency by asking: “Do you consider acting a short-cut over animation? I think the best near-term way to view this is as a way of opening the medium of animation to the lively arts — like juggling, martial arts, and acting in general. There's no reason that we can't incorporate the talents of realtime performers as well as the painstaking work of gifted animators. It may not necessarily be cheaper or faster; it depends on what actor you're trying to shoot. But it certainly expands the scope of the medium.”
Williams believes that videogame animators are likely to recognize the advantages of performance capture, especially given their widespread reliance on motion capture. And he notes that it could be used to have actors drive the performances of animatronic characters. After all, he remarks, “For its theme parks, Disney developed actual performance capture in the 1950s to drive audio animation.”
But Williams wonders about resistance among animated moviemakers. “In animated movies, it's amazing how strongly the artists regard their traditions. Pixar's John Lasseter emphasizes that his fellow animators are all actors. “With a technology like ours, they'd have a chance to prove it — and go head to head with the likes of Judi Dench.”
During his extensive career, which includes tenures at Apple Computer and DreamWorks, Williams has experimented with facial animation techniques before. But the strategy of applying computer vision techniques to animation has moved his research into a new arena. “I think the time is right for the useful application of these technologies in motion pictures. They may not be ready to perform surgery, but they're safe enough to work on cartoons!”
Williams ultimately sees the Human Face Project as the continuation of a venerable tradition. “At the dawn of motion pictures, Eadweard Muybridge was interested in photography as a tool for capturing data. It only secondarily was taken up by others as a sensational entertainment medium. But it's always advanced the purposes of science, and it will continue to do so until the last producer has choked on his last cigar!”
Continue the discussion on “Crosstalk” the Millimeter Forum.