Author: Alex Shipps | MIT CSAIL<\/p>\n
Whether you\u2019re describing the sound of your faulty car engine or meowing like your neighbor\u2019s cat, imitating sounds with your voice can be a helpful way to relay a concept when words don\u2019t do the trick.<\/p>\n
Vocal imitation is the sonic equivalent of doodling a quick picture to communicate something you saw \u2014 except that instead of using a pencil to illustrate an image, you use your vocal tract to express a sound. This might seem difficult, but it\u2019s something we all do intuitively: To experience it for yourself, try using your voice to mirror the sound of an ambulance siren, a crow, or a bell being struck.<\/p>\n
Inspired by the cognitive science of how we communicate, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers have developed an AI system that can produce human-like vocal imitations with no training, and without ever having “heard” a human vocal impression before.<\/p>\n
To achieve this, the researchers engineered their system to produce and interpret sounds much like we do. They started by building a model of the human vocal tract that simulates how vibrations from the voice box are shaped by the throat, tongue, and lips. Then, they used a cognitively-inspired AI algorithm to control this vocal tract model and make it produce imitations, taking into consideration the context-specific ways that humans choose to communicate sound.<\/p>\n
The model can effectively take many sounds from the world and generate a human-like imitation of them \u2014 including noises like leaves rustling, a snake\u2019s hiss, and an approaching ambulance siren. Their model can also be run in reverse to guess real-world sounds from human vocal imitations, similar to how some computer vision systems can retrieve high-quality images based on sketches. For instance, the model can correctly distinguish the sound of a human imitating a cat\u2019s \u201cmeow\u201d versus its \u201chiss.\u201d<\/p>\n
In the future, this model could potentially lead to more intuitive \u201cimitation-based\u201d interfaces for sound designers, more human-like AI characters in virtual reality, and even methods to help students learn new languages.<\/p>\n
The co-lead authors \u2014 MIT CSAIL PhD students Kartik Chandra SM \u201923 and Karima Ma, and undergraduate researcher Matthew Caren \u2014 note that computer graphics researchers have long recognized that realism is rarely the ultimate goal of visual expression. For example, an abstract painting or a child\u2019s crayon doodle can be just as expressive as a photograph.<\/p>\n
\u201cOver the past few decades, advances in sketching algorithms have led to new tools for artists, advances in AI and computer vision, and even a deeper understanding of human cognition,\u201d notes Chandra. \u201cIn the same way that a sketch is an abstract, non-photorealistic representation of an image, our method captures the abstract, non-phono–<\/em>realistic ways humans express the sounds they hear. This teaches us about the process of auditory abstraction.\u201d<\/p>\n The art of imitation, in three parts<\/strong><\/p>\n The team developed three increasingly nuanced versions of the model to compare to human vocal imitations. First, they created a baseline model that simply aimed to generate imitations that were as similar to real-world sounds as possible \u2014 but this model didn\u2019t match human behavior very well.<\/p>\n The researchers then designed a second \u201ccommunicative\u201d model. According to Caren, this model considers what\u2019s distinctive about a sound to a listener. For instance, you\u2019d likely imitate the sound of a motorboat by mimicking the rumble of its engine, since that\u2019s its most distinctive auditory feature, even if it\u2019s not the loudest aspect of the sound (compared to, say, the water splashing). This second model created imitations that were better than the baseline, but the team wanted to improve it even more.<\/p>\n To take their method a step further, the researchers added a final layer of reasoning to the model. \u201cVocal imitations can sound different based on the amount of effort you put into them. It costs time and energy to produce sounds that are perfectly accurate,\u201d says Chandra. The researchers\u2019 full model accounts for this by trying to avoid utterances that are very rapid, loud, or high- or low-pitched, which people are less likely to use in a conversation. The result: more human-like imitations that closely match many of the decisions that humans make when imitating the same sounds.<\/p>\n After building this model, the team conducted a behavioral experiment to see whether the AI- or human-generated vocal imitations were perceived as better by human judges. Notably, participants in the experiment favored the AI model 25 percent of the time in general, and as much as 75 percent for an imitation of a motorboat and 50 percent for an imitation of a gunshot.<\/p>\n Toward more expressive sound technology<\/strong><\/p>\n Passionate about technology for music and art, Caren envisions that this model could help artists better communicate sounds to computational systems and assist filmmakers and other content creators with generating AI sounds that are more nuanced to a specific context. It could also enable a musician to rapidly search a sound database by imitating a noise that is difficult to describe in, say, a text prompt.<\/p>\n In the meantime, Caren, Chandra, and Ma are looking at the implications of their model in other domains, including the development of language, how infants learn to talk, and even imitation behaviors in birds like parrots and songbirds.<\/p>\n The team still has work to do with the current iteration of their model: It struggles with some consonants, like \u201cz,\u201d which led to inaccurate impressions of some sounds, like bees buzzing. They also can\u2019t yet replicate how humans imitate speech, music, or sounds that are imitated differently across different languages, like a heartbeat.<\/p>\n Stanford University linguistics professor Robert Hawkins says that language is full of onomatopoeia and words that mimic but don\u2019t fully replicate the things they describe, like the \u201cmeow\u201d sound that very inexactly approximates the sound that cats make. \u201cThe processes that get us from the sound of a real cat to a word like \u2018meow\u2019 reveal a lot about the intricate interplay between physiology, social reasoning, and communication in the evolution of language,\u201d says Hawkins, who wasn\u2019t involved in the CSAIL research. \u201cThis model presents an exciting step toward formalizing and testing theories of those processes, demonstrating that both physical constraints from the human vocal tract and social pressures from communication are needed to explain the distribution of vocal imitations.\u201d<\/p>\n Caren, Chandra, and Ma wrote the paper with two other CSAIL affiliates: Jonathan Ragan-Kelley, MIT Department of Electrical Engineering and Computer Science associate professor, and Joshua Tenenbaum, MIT Brain and Cognitive Sciences professor and Center for Brains, Minds, and Machines member. Their work was supported, in part, by the Hertz Foundation and the National Science Foundation. It was presented at SIGGRAPH Asia in early December.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"