Prof. Ronnie Wilbur (left) of Purdue’s speech, language and hearing science department joined forces with Asst. Prof. Nicoletta Adamo-Villani of the school’s computer graphics technology three years ago to help deaf children.

Such a system is an effective teaching tool because it gives the students control over the environment, offers self-paced and repetitive learning, presents concepts in a concrete rather than abstract manner, features an environment that feels safe, and, as a game, is motivational. All of these characteristics help the student overcome challenges that are common to people with learning disabilities.

The Animation Process
The game features four stores in which students learn a different math concept: a bakery where they learn about weight, a clock store where they learn to tell time, a toy store where they learn how to count money and a candy store where they learn addition and subtraction. The avatar and the child can communicate; for example, one of the avatars can ask a question and the child can sign a simple answer, such as a number, or the child can pick up a certain number of candies and the avatar can make the sign for that number.

Motion-capture is used to record the signs, so the avatars — characters such as rabbits, robots and pigs that serve as the storekeepers — can communicate with the students in real time; the remaining animation, including facial expressions, is key-frame. All models were created in Maya.

In order for the Mathsigner characters to be displayed in the FLEX system, the polygon count had to be reduced to maintain the fast frame rate and real-time interaction, and new facial rigs had to be created. Polygons are concentrated in areas where detail is needed most, namely parts of the body that bend or twist. This keeps the signing realistic-looking.

“It requires an extremely fluid motion,” says Adamo-Villani. “We’re working with signers who are very picky. The motion has to be very realistic.” At the same time, the animation look is colorful, cartoony and stylized. “We’re using Disney’s Toontown as inspiration,” she explains, noting that while many of the challenges of this project are similar to any animation production, a unique challenge is to make the stylized characters do the signing motions organically, in real time.

The project is popular with students in computer animation at Purdue. They see it as a good opportunity to learn both motion capture and key-frame animation, and they end up with a strong portfolio piece. Graduate, masters and undergraduate students are working on it, led by David Jones, a graduate student in the department of computer graphics technology.

The game features four stores in which students learn a different math concept, including a bakery where they learn about weight and a clock store where they learn to tell time.

The application is being refined to add more gaming elements. Currently, users can go into individual stores, one at a time. The creators are looking at ways to link the stores so the child can move from one to another as he or she completes a task, making the action more like a videogame.

Feedback from children led the animators to add more storytelling as well. “When the kids were testing it, we got comments like, ‘Why don’t you have a story?’“ says Adamo-Villani, who previously worked on the Ironman television series at Marvel, as well as in computer graphics research at the University of California-Riverside.

Some improvements are under consideration on the technology side, too. The researchers plan to look into using more sensitive gloves, such as those employed for mo-cap, which would enable more complex signing by the children as they use the immersive system. In addition, the current FLEX system is expensive and must be used in a special room, which makes it impractical for both institutions and individuals. The researchers are exploring different options for a portable yet still immersive system. “That’s one of the challenges of the project,” Adamo-Villano says. “The technology’s not there yet.”

For more information visit Mathsigner at www2.tech.purdue.edu/cgt/I3/ and SMILE at www2.tech.purdue.edu/cgt/I3/SMILE/.

Karen Raugust is a Minneapolis-based freelance business writer specializing in animation, publishing, licensing and art. She is the author of The Licensing Business Handbook (EPM Communications).