Author: Adam Zewe | MIT News<\/p>\n
The crack of a bat accompanied by a roar from the crowd can capture a baseball fan\u2019s attention, even if they miss seeing the home run. Humans may be visual creatures, but we also perceive the world around us by listening to it.\u00a0<\/p>\n
To improve artificial perception, MIT Professor Paris Smaragdis SM \u201997, PhD \u201901 seeks to make machines that can hear and understand sound even better than humans.<\/p>\n
A machine with super-human hearing could rapidly pinpoint the location of a traffic accident by listening for squealing tires and crunching metal, or identify when an industrial drill might break while it is boring through dense bedrock to dig a geothermal well.<\/p>\n
\u201cI like to say that I do the same thing the computer vision folks do, but instead I do it for hearing,\u201d says Smaragdis, a professor with joint appointments in the Music and Theatre Arts program of the School of Humanities, Arts, and Social Sciences, and the Department of Electrical Engineering and Computer Science in the School of Engineering and the MIT Schwarzman College of Computing.\u00a0<\/p>\n
His research group explores areas like machine learning for sound source separation, smart devices that can augment the audio surroundings of humans, and better tools to help people understand, manipulate, analyze, and create music.<\/p>\n
\u201cHaving had a musical background, I\u2019ve been interested in examining these ideas. What does it take to make machines understand sound? There is a huge domain of applications for machines that can listen,\u201d says Smaragdis, who joined MIT in 2025.<\/p>\n
A musical prelude<\/strong><\/p>\n Smaragdis, who grew up in Greece, began playing classical guitar at a young age. He enjoyed playing traditional pieces but reached a turning point in his musical journey when his father brought home a small synthesizer.<\/p>\n \u201cThat little synthesizer had a function that allowed you to make your own sounds, and I became completely captivated by the possibilities,\u201d he says.<\/p>\n Synthesizers were taking a more prominent place in pop music during the 1980s, adding to his fascination with the technology.<\/p>\n As he advanced through school, Smaragdis wanted to study music made with synthesizers. But this was not an educational path offered by the more traditional European conservatories at the time.<\/p>\n Instead, he enrolled at Berklee College of Music in Boston, into a program where he could explore the use of computers and synthesizers to shape and manipulate sound.<\/p>\n \u201cWith computers and synthesizers, I could make new sounds that have never been heard before,\u201d he says.<\/p>\n At the same time, Smaragdis became immersed in jazz fusion. He declared bass as his major at Berklee since it would allow him more opportunities to play with jazz ensembles.<\/p>\n \u201cI found the bass to be sonically more interesting than the guitar. It was a more unexplored domain, and I\u2019ve always gravitated toward unexplored domains,\u201d he says.<\/p>\n That interest in the unexplored carried over into his coursework. He and mentor Richard Boulanger, a professor of electronic production and design, created new tools to help people use computers to write, produce, and enhance music.\u00a0<\/p>\n Through his studies at Berklee, Smaragdis embarked on a collaborative project with Professor Barry Vercoe (who\u00a0passed away in 2025<\/a>) at the MIT Media Lab to create software synthesizers for music creation. Building on that project, Smaragdis pursued graduate studies in Vercoe\u2019s group, where he completed his master\u2019s degree and PhD, followed by a postdoc.<\/p>\n \u201cIn the Media Lab, I got exposed more to the computer part of computer music. That\u2019s where my attention shifted a bit. Now I really wanted to make computers that could understand sound,\u201d he says.<\/p>\n Tuning in to unexplored domains<\/strong><\/p>\n At that time, many students in the Media Lab were using signal processing techniques to help machines handle sound. Smaragdis chose a less explored approach \u2014 machine learning \u2014 to separate himself from the pack.<\/p>\n He found that training machine-learning models to understand sound offered advantages that couldn\u2019t be attained with traditional signal processing techniques.<\/p>\n \u201cOver time, it became really clear to me that we need to teach computers how to listen. We can\u2019t just tell them how to listen. If anything, we don\u2019t even know how we\u00a0<\/em>humans listen,\u201d he says. \u201cWe don\u2019t have a vocabulary to explain what it means to listen, or how to tell the difference between a cat\u2019s meow and the sound of a violin.\u201d<\/p>\n After finishing his postdoc, Smaragdis began a career at Mitsubishi Electric Research Labs, where he worked on artificial intelligence systems that could scan video soundtracks and understand what is happening on screen.\u00a0<\/p>\n \u201cWhat if you record an entire soccer game, but you only want to watch the highlights? In computer vision, it is very difficult to identify what a highlight is. But on the other hand, if you only listen to the sounds, then when the crowd goes crazy you would know that something interesting is happening,\u201d he says.<\/p>\n That system was eventually incorporated into electronic devices, including a personal video recording system Mitsubishi was developing at the time, and into traffic monitoring systems and cell phones.<\/p>\n Smaragdis later joined Adobe, where he created the audio research team that built user-friendly tools to help artists, musicians, and producers manipulate sound without needing a lot of expertise.<\/p>\n \u201cAs we did that work, I kept coming up with deeper technical problems that needed to be resolved. I jumped into academia to address that itch,\u201d he says.<\/p>\n Smaragdis joined the faculty at the University of Illinois at Urbana-Champaign, where he and his group incorporated the latest machine-learning techniques into audio-processing systems.<\/p>\n One of the technologies they developed utilized neural networks for separating individual sounds from audio mixtures. That project caught the attention of the team working on the 2021 documentary mini-series \u201cThe Beatles: Get Back.\u201d<\/p>\n The production team enlisted Smaragdis to help clean up some archival Beatles recordings.<\/p>\n \u201cThat was a very different way of audio processing than what I was doing in academia. Now, I had to do something that sounded good but didn\u2019t necessarily have to look good on paper. We didn\u2019t have a formula we could optimize,\u201d he says.<\/p>\n Every recording involved different challenges. For instance, in one clip John Lennon was playing music while Paul McCartney was horsing around, reading headlines from newspapers and making fun of George Harrison.<\/p>\n \u201cWe were trying to come up with a technological solution that could help us do everything, as opposed to addressing each problem on its own,\u201d he says.<\/p>\n His contributions helped the team win an Emmy Award for sound editing for a nonfiction or reality program, in 2022.<\/p>\n A concerted effort<\/strong><\/p>\n After 15 years at the University of Illinois, Smaragdis decided to join the faculty at MIT to tackle the thorny, interdisciplinary problems that were emerging in the field.\u00a0<\/p>\n His group now focuses on two main areas. For one, they develop systems that make computers more useful tools for musicians.\u00a0<\/p>\n One project in this area is enabling machines to understand how to listen to music, so they can provide constructive feedback to musicians.<\/p>\n \u201cWe want to enhance the toolbox a musician has. We don\u2019t want to use computers to replace musicians,\u201d he says.<\/p>\n On the other side, his group focuses on the fundamental tools scientists and artists alike can use to process, analyze, and manipulate audio.<\/p>\n For instance, his team has recently been exploring algorithms that allow sophisticated signal processing where traditional frames of reference, such as sampling rates, frame sizes, or context-windows, are not fixed \u2014 or even defined.\u00a0<\/p>\n That implies different computational structures that are yet unexplored, despite their potential for more efficient processing, Smaragdis says.<\/p>\n \u201cThat requires rethinking a lot of the basics, but once you have that ability, you can start getting much more flexible and more powerful systems,\u201d he adds.<\/p>\n They also explore ways to incorporate the latest techniques, like large language models, into speech recognition and audio enhancement.<\/p>\n \u201cAI is an immensely powerful tool, but I don\u2019t think we have found the ideal way to use it for sound and music yet,\u201d he says.<\/p>\n He also enjoys mentoring students and helping them materialize their visions for research projects.<\/p>\n Since few of his students have experience using computation in sound, he emphasizes hands-on lessons so they see and hear how to transform equations into something tangible.<\/p>\n Smaragdis still feels a thrill from making music, whether with a computer or a more traditional instrument. Off campus, he tries to play jazz bass whenever he can find the time, and enjoys exploring the rich musical community of Boston with his wife and 15-year-old son.<\/p>\n Outside the musical realm, he plays and coaches underwater hockey \u2014 a sport where players wear diving masks and use sticks to push a heavy puck across the bottom of a pool toward their opponent\u2019s goal. He coached the underwater hockey team at the University of Illinois, and is hoping to revive the MIT underwater hockey club soon and start hosting tournaments on campus.<\/p>\n While underwater hockey may be a lesser-known sport, it fits with Smaragdis\u2019 lifelong interest in pursuing the underexplored.<\/p>\n \u201cI\u2019ve always like doing things that were different from what everyone else was doing. If you can create a new sound that no one has heard before, what does that mean about your ability to create new things? I still find these questions completely fascinating,\u201d he says.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"