Researchers work with the military to create the next generation of training technology
Bang! Bang! Bang! comes the knock on the door. Through a window on the other side of the room you see the streetscape of an Iraqi city, complete with helicopters flying overhead and rockets landing in the street. When rockets hit, the room rumbles. It’s only natural to feel anxious, alert, and a little afraid.
The window, however, is merely a computer projection on a screen, surrounded by dressed-up plywood walls that you might find on a movie set. The door is real, and the knocking is startlingly urgent, as if someone is really banging on the wood. The ceiling overhead is perforated with blast holes; the furniture is smashed; and the sounds of artillery fire and an Islamic call to prayer, as if issued from a loudspeaker, can be heard in the distance. When the door opens, a man in a mask – a figure projected on a second screen – yells something, then appears to spray the room with bullets. A digital projector pockmarks a far wall with bullet holes.
The elaborate marriage of stagecraft and technology is part of a project called Flatworld, which takes its name from the scenery pieces, often called "flats," that compose the background on a theater stage or a film set.
William R. Swartout, director of technology at the Institute for Creative Technologies at the University of Southern California, points to the ground in front of the terrorist at the door – the digital man is casting a shadow.
"Shadows might seem like a frill," he says. "But people who do these types of staging exercises say that you should always watch for shadows. You can see the shadow before you see the person. If you just see the person and he’s got a gun, it’s too late.
"That’s one reason why we are trying to push the edge of making it realistic here, so people can take advantage of the full set of sensory inputs."
This is one of many projects under development at the Institute for Creative Technologies, a research group that is supported primarily by the U.S. Army. At ICT, as the institute is called, researchers in simulation, language-recognition, and animation technology work with people from the entertainment industry to devise new, realistic methods for training soldiers using virtual reality. These are the high-tech dress rehearsals for future theaters of war.
The U.S. military is increasingly interested in immersive virtual reality, gaming technology, and computer-based simulations that can mimic situations in the real world. Professors with an eye to the future of the classroom should take note: Many technologies developed for the military have found their way into civilian use, with the Internet being perhaps the most prominent example.
In the future, one can imagine a version of Flatworld being used to train emergency-response teams. In fact virtual-sound researchers at Southern Cal who work with ICT are experimenting with high-fidelity sound systems to recreate the deafening and disorienting atmosphere of a burning building, with the goal of training firemen.
Mr. Swartout sees applications for Flatworld even in settings as benign as museums. "You could do things you couldn’t do with just the artifacts," he says. "Put someone on the Titanic, or actually experience what it’s like to walk down a street in ancient Rome. You’re not just looking at the urns and vases. You’re actually there."
Less collateral damage
Simulators are already widely used to train soldiers to pilot aircraft and drive ground vehicles, and some of those can rumble and bounce as if they were really on a rutted desert road. Video games like America’s Army and Full Spectrum Warrior, which was developed by ICT for the military, are not just shooter games, but incorporate real military tactics in play. In virtual environments, soldiers can train on simulated weapons that, thanks to pneumatic components, have the kick and bang of real guns.
John A. Parmentola, director for research and laboratory management for the Army, says much of the military’s interest in virtual reality stems from the liabilities of war games: Live training, which can entail moving a lot of heavy equipment or personnel, is expensive. And it can be dangerous.
It can also have a negative impact on the environment, according to experts who have criticized military training efforts. Obviously, in a virtual or simulated setting, valuable gas isn’t burned, hazardous explosives aren’t used, and big tanks don’t roll over the breeding grounds of endangered species.
But simulators, games, and virtual reality offer benefits even beyond those, says Jeff Wilkinson, ICT program manager at the Army’s Simulation & Training Technology Center. Simulated training can be standardized, and soldiers’ progress can be logged. When soldiers make mistakes, instructors can easily take them back to a crucial moment in the training to hammer home a concept or to point out an error.
In another simulation being developed here, participants walk into a ragged clinic somewhere in war-torn Iraq. You immediately get a sense of the dire situation. You hear sobbing, and to your right, you see a woman crying over a body covered with a blood-soaked sheet. Flies are buzzing ravenously around it.
The clinic’s doctor, his white coat smudged with dirt and blood, stands in his office near the back. Your orders are clear: Persuade him to move the clinic somewhere else. A military operation is moving through this neighborhood soon, and he and his patients aren’t safe.
You and the doctor exchange pleasantries. Then you state your case: "I want to move this clinic to a safer location."
"This conflict is madness," the doctor shoots back. "It’s killing people." Classic avoidance strategy.
A Unique Combination
But the clinic exists only virtually, on a wide-screen display. And the good doctor is merely a digital avatar, programmed for argument and barter.
Such training in social interactions has been a particular strength of the institute, says Jeremy Bailenson, an assistant professor of communication at Stanford University, who studies humans’ reactions to virtual environments.
"ICT’s strength is not so much in making a better battle simulation, but in taking more social types of interactions, which are crucial in any campaign, and training soldiers on that," he says.
"There may be some who are doing natural-language processing better than [the institute], and there may be some doing graphics better than them, but there is not an institution in the world that has put together all of the small pieces of social interaction in a psychologically meaningful way like they have."
The virtual negotiation tool relies on programming that goes beyond sharp graphics and complex language-recognition technology. Mr. Swartout interrupts a demo of the negotiation game to show what is going on in the doctor’s head. The computer is not only processing the overtures of the human user, but is also tracking various emotions of the doctor, such as his joy and his trust.
"Trust turns out to be one of the critical variables in a successful negotiation," Mr. Swartout says. "If you don’t have enough trust, it goes badly."
The doctor follows a model laid out by psychologists who study patterns of negotiation. First the doctor avoids the conversation; then he views the negotiation as a win-lose proposition; and, finally, he enters real bartering with the soldier-player. At any point, of course, he can cut off talks if he is not satisfied with a player’s responses or offers.
The game may be more difficult in the future, as researchers at the institute are working on technology that will track a human player’s body language, facial expressions, and tone of voice. A sweet talker with an unconvincing body posture might have trouble with the mission.
It is easy to see how technology like this might fit in a more traditional classroom setting someday. Perhaps business schools will have students land deals with virtual CEOs, or law students might cross-examine virtual witnesses. Mr. Swartout says that Southern Cal’s Marshall School of Business has already expressed interest in the project.
"I see what we are doing as laying down some of the basic research groundwork that is going to ultimately enable new kinds of experiences that will be useful in both education and entertainment," he says. The virtual negotiation project could "open up a whole new genre of computer games and forms of entertainment, where the emphasis is much more on relationships, emotions, and story rather than action adventure."
The institute has a flair for Hollywood-style storytelling in its projects, some of which even rely on moviemaking techniques. One research project, Army Excellence in Leadership, uses interactive films and case-study techniques to convey some of the intangibles of leadership skills.
In a film called Power Hungry, Capt. Michael Young’s brash leadership style and inability to read the locals leads to mayhem during a food-distribution mission in Afghanistan.
A Hollywood screenwriter wrote the script. His touch can be seen in the film’s characters, such as Omar, the antagonistic Afghan warlord, who carries himself with the sort of poise seen among bad guys in James Bond films. (His hovering henchmen pop open a fold-out chair when he appears ready to sit down.) Captain Young, meanwhile, puts on a show of confidence and authority. He yells at his executive officer and ignores the advice of more-experienced soldiers, and he is in over his head by the short film’s end, when locals storm the food trucks.
If the take-home points are not clear by the time the credits roll, viewers can mark parts of the film and engage the characters in question-and-answer sessions on a Web page. Artificial intelligence recognizes typed-in questions from the students, like "How would you describe your leadership style?"and pairs them with prerecorded responses from the actors, in character. (Captain Young, ever confident, will say that he established a climate of "openness, of two-way communication.")
"I don’t see this as a substitute for human instructors," but as an aid, says Michelle L. Zbylut, a research psychologist with the Army Research Institute for the Behavioral and Social Sciences, which helped produce the interactive films. "One of the problems is that instructors have to teach more in a shorter period of time, so this can help with preparation." The storytelling format, Army officials say, leads students to retain more of the major points of the lessons than they would from a case study on paper or PowerPoint.
Training like this is also highly portable, and can even be run on laptops, says Mr. Wilkinson, of the Army’s Simulation & Training Technology Center. Portability provides more opportunities to train individual soldiers in leadership and cultural issues. As the Abu Ghraib scandal and the kidnapping and murder cases involving Marines in Iraq have shown, "a single soldier can set off events with great consequences," Mr. Wilkinson says.
Sensory Experience
Other laboratories and institutions are also trying to harness the senses in training environments. Glenn A. Martin, a senior research scientist at the University of Central Florida’s Institute for Simulation and Training, which works extensively with the military, says that he is considering the use of virtual smells in some of the training devices he is creating for medics. He says he and colleagues are pondering whether smelling blood and guts would be good or bad for medic training.
"I would think that for a true combat medic, you want to desensitize them to that," he says. "You don’t want them freaking out when they are out in the field."
A good training environment relies on a certain level of believability, experts say. New technologies under development, here at Southern Cal and at other universities and laboratories, are striving for a virtual reality known only in science fiction.
"The holy grail is the holodeck," the holographic virtual-reality system seen in Star Trek episodes, Mr. Parmentola says. Other goals for researchers include advances in virtual touch (also known as haptics) and programs that can simulate human interactions, like conversation and negotiation, at a more sophisticated level.
"We are still pretty far from the chaos of the battlefield," says Mr. Bailenson, of Stanford. But a virtual environment "doesn’t have to visually match the beauty and insanity of the real world to create psychological connections with characters."
By SCOTT CARLSON
this article was taken from the Chronicle of Higher Education