Purdue’s project to use computer animation and VR to teach deaf students math includes Mathsigner, an interactive PC game. All images courtesy of the Computer Graphics Technology Department at Purdue University, © Nicoletta Adamo-Villani.

Two Purdue University professors are studying how to use computer animation and virtual reality to teach deaf children in grades K through 4 about mathematics principles, and to communicate math terminology using American Sign Language. The reason for the project is that, for deaf children, reading instruction is delayed; therefore most kids in this age group cannot use traditional textbooks. They also aren’t able to take full advantage of secondary learning opportunities that non-disabled kids have, such as television and dinner table conversation, and their parents may not be fluent enough in sign language to teach them about math.

The Purdue project hopes to alleviate these challenges by allowing deaf children to learn math concepts from animated characters, who use sign language to communicate with the kids in a computer-game setting. Even more effective is extending the program into virtual reality, where the characters and the children can communicate with each other in sign language in real time, in a fully immersive environment.

SMILE, the project’s VR game, features a robot virtual signer and a virtual clock store. The game is projected onto three walls and the floor of a special room using Fakespace Labs’ FLEX.

Three years ago, Nicoletta Adamo-Villani, assistant professor in Purdue’s Department of Computer Graphics Technology, who teaches 2D and 3D animation and specializes in character animation and design, joined with Ronnie Wilbur, professor and chair of linguistics in Purdue’s Department of Speech, Language and Hearing Science. The first phase of their joint project focused on creating an interactive PC game called Mathsigner, in which 3D avatars communicate with the user in sign language. The team recently earned a grant from the National Science Foundation to proceed with this aspect of their work; it is currently in prototype form and in use at the Indianapolis School for the Deaf as a web/CD-deliverable desktop application.

A more recent phase of the project extends the software into an immersive, virtual-reality world. Working with Purdue’s Envision Center for Data Perceptualization, the researchers are creating a game that is projected onto three walls and the floor of a special room, using Fakespace Labs’ FLEX system. (The VR game can be played on a PC as well but it is not immersive in that format.) When using the application, the child wears a special headset, lightweight stereoscopic glasses, pinch gloves and a wrist tracker. These devices allow the child to communicate with the avatars and redraws the animation to follow the child’s perspective as he or she moves.

“Research shows that immersive games are more effective than non-immersive games,” says Adamo-Villani. Studies have found that humans process visual information 60,000 times faster than text-based information, she reports, noting that students can raise their math scores 16% after eight weeks using an immersive VR program.