杏吧原创

In different cultures, a smile can mean anything from aggression to embarrassment 鈥� now Tinseltown special effects are helping soldiers avoid face-offs

WHETHER we like it or not, our faces can speak more eloquently than mere words. Poker players or diplomats might bluff their opponents with an impassive mien, but most of us aren鈥檛 such good actors. With us the mask continually slips, betraying feelings that our more easily controllable speech does not.

Our digital creations have the opposite problem: their faces are not expressive enough. The virtual environment of a video game might be vivid and credible, but its inhabitants鈥� visages will be bland and emotionless: there is little or nothing there for the player to interpret or empathise with. That may be just as well if the aim of the game is to blow your opponent鈥檚 brains out, but for many applications it is a serious drawback.

Things are set to change, though, and not just for games. In movies such as X-Men: The last stand and the upcoming Tron: Legacy, the facial movements of grey and wrinkled actors are mapped onto younger versions of themselves. And the blockbuster Avatar created impressively realistic digital creatures based on information captured by tracking the movement of 52 green dots painted onto the faces of actors.

In this area, even the big bucks of the big screen are being eclipsed by investment from the military. 鈥淣inety-five per cent of our budget comes from the US Department of Defense,鈥� says , director of the University of Southern California鈥檚 Institute for Creative Technologies (USC-ICT) in Playa Vista, which specialises in avatar design.

Virtual peacekeeping

Why so? Because US military operations are increasingly about not just guns and gung-ho, but also building bridges and nations. The success or failure of such missions hinges on soldiers engaging with people in unfamiliar cultural settings. That requires them to have an exact sense of what might lie behind their negotiating partners鈥� masks. And one way of getting that knowledge across is to use realistic, immersive virtual-reality programs in which the soldiers are trained by being presented with avatars that behave in a similar way to the people they will meet in the field.

That, however, is easier said than done. 鈥淲hen two people converse, the speaker influences the listener and the listener influences the speaker, so you have a constant dynamic in action, with a lot of subtle expressions that are influencing that interaction,鈥� says , a computer scientist at USC-ICT. If the avatar鈥檚 facial expressions do not react appropriately to ours, we do not act as we do with real humans.

The effect this has on our behaviour can be huge, too. Last year, of USC-ICT and two other researchers asked people to play a computer version of the prisoner鈥檚 dilemma. Beloved of game theorists, it involves two people who play the role of partners in crime who have been captured. Each one is given the choice of either betraying their colleague or staying silent 鈥� without knowing what the other will do. If each betrays the other, they both go to prison for a year. If both stay silent, they both go down 鈥� but for only three months. If one betrays and the other stays silent, the one who is betrayed goes to prison for three years while the betrayer goes free.

Under such rules, the rational strategy is always to betray your colleague, because that gives you the better outcome whichever option they choose. Yet hundreds of studies over the years have shown that humans do not automatically do this. Cooperative behaviour is far more common among us than it should be if we were behaving rationally ().

Gratch and his colleagues had people play this game repeatedly against two varieties of avatar. One sort showed cooperative emotions: for example, seeming grateful for mutual cooperation or ashamed if it betrayed a partner who had cooperated. The other sort exhibited selfish emotions that showed they were only out to optimise their own outcome: they were jubilant when they betrayed an opponent who had cooperated, sad when they cooperated only to discover their opponent had betrayed them, and so on.

Intriguingly, even though all the avatars were programmed to act in the same way, betraying in response to betrayal and collaborating in response to collaboration, the human participants behaved cooperatively with the avatars that looked as if they wanted to cooperate. Other studies paint a similar picture of our susceptibility to 鈥渉uman鈥� influence, showing for instance that we learn more effectively when taught by an avatar capable of expressing different emotional states (). Hence the demand for more realistic avatars.

There is just one small problem: to work out what it is avatars should be doing with their faces, we need to understand what we ourselves do, and that can be complicated. Take nodding, for example. We might regard it simply as a rather lazy way to signify assent or understanding, but the reality is more complex. If someone nods at the wrong times, or too often, we quickly recognise this as a signal of boredom or a desire for the conversation to end.

What are appropriate cues for a nod when two Americans converse? Lowered vocal pitch, pauses and eye gazes all seem to be, but what combination and timing of these signals will predictably elicit a nod? To find out, Morency and his colleagues used software to analyse recordings of 50 people describing to 50 listeners what they had just seen in a video. Every time the program detected nodding in the listener, it went searching for visual and verbal cues coming from the speaker.

The result was an expanded list of cues, some purely verbal 鈥� the words 鈥渢hat鈥� and 鈥渁nd鈥� cropped up often 鈥� and a list of cue combinations that were highly likely to elicit nods. 鈥淚f someone was looking away, but then looked at the listener and used the word 鈥渁nd鈥� followed by a pause, this almost always triggered a head nod in response,鈥� says Morency. When it didn鈥檛, often a verbal 鈥淥K鈥� came instead.

It鈥檚 not just nodding that we need to understand. Where we look during conversation, and how we use facial expressions to modify what we say, can also be highly significant. 鈥淔or example, you could say something nasty but smile to modify it,鈥� says Gratch. Not only that, but different cultures evaluate facial gestures in different ways. Smiling may soften harsh words in America, but it is not a universal expression of friendliness. 鈥淚n Arab cultures, for example, it is an aggressive display and in Japanese culture a display of embarrassment,鈥� says Gratch. Similarly, Americans do not like silence very much and will jump in to fill gaps in conversation, whereas in Arab cultures people are happy to let silence linger.

The USC-ICT researchers are feeding a detailed analysis of such cultural behavioural differences 鈥� nods and smiles, pauses and eye gazes and also gesticulation 鈥� into a training environment known as , to be used by soldiers being deployed to Iraq. The emotional believability of a Hollywood avatar, so the thinking goes, should help soldiers become sensitive to cultural issues in the way they dress, behave and raise delicate subjects.

鈥淭he game teaches soldiers to learn to take off helmets and sunglasses before trying to speak with Iraqis and even then, not rushing into the conversation without engaging in small talk and sipping some tea first,鈥� says John Hart of the US Army Research, Development and Engineering Command. Do the wrong thing, and the avatars use not just words but also gestures, eye contact and expressions appropriate to their culture to convey their emotional state.

It seems to work. In three hours, the new environment allows soldiers to achieve the same proficiency in cultural interaction as they would achieve from six to eight hours of interaction using traditional methods. At the moment, however, learners must indicate their actions 鈥� 鈥淚鈥檓 taking off my helmet鈥�, 鈥淚 smile鈥� 鈥� through keyboard or mouse inputs. True interactivity requires the avatar itself to recognise such actions and expressions without being told, and react accordingly.

That too is in the works. Drawing on expertise USC-ICT researchers developed in building avatars for movies such as Avatar, Morency and his team are using cameras and microphones to record facial expressions, analyse them and translate them into an avatar鈥檚 reaction. 鈥淲hile our team has a keen understanding of when to show certain expressions, the people who worked on Avatar are the masters of how to artificially create them,鈥� he says. The first stop is teaching the software the subtleties of nodding, with facial expressions of confusion and frustration to follow.

Military reality and Tinseltown fantasy seem to be heading for an unlikely partnership. 鈥淲e want to move towards the Star Trek holodeck,鈥� says Hart, referring to the reality simulators featured in the TV dramas, 鈥渨here the technology being created can give us believable virtual people for soldiers to fully interact with.鈥� Once that technology is there, it is only a matter of time before we are all facing up to a new digital world of emotion in games and films.

鈥淢ilitary reality and Tinseltown fantasy seem to be heading for an unlikely partnership鈥�