Author: Matilda Bathurst | Arts at MIT<\/p>\n
Sketch a doodle of a drum or a saxophone to conjure a multi-instrumental composition. Look into a webcam, speak, and watch your mouth go bouncing across the screen \u2014 the input for a series of charmingly clunky chain reactions.<\/p>\n
This is what visitors to the MIT Lewis Music Library encounter when they interact with two new digital installations, \u201cDoodle Tunes\u201d and \u201cSounds from the Mouth,\u201d created by 2022-23 Center for Art and Technology (CAST) Visiting Artist Andreas Refsgaard<\/a> in collaboration with Music Technology and Digital Media Librarian Caleb Hall. The residency was initiated by Avery Boddie, Lewis Music Library department head, who recognized Refsgaard\u2019s flair for revealing the playfulness of emerging technologies. The intricacies of coding and machine learning can seem daunting to newcomers, but Refsgaard\u2019s practice as a creative coder, interaction designer, and educator seeks to open the field to all. Encompassing workshops, an artist talk, class visits, and an exhibition, the residency was infused with his unique sense of humor \u2014 a combination of lively eccentricity and easygoing relatability.<\/p>\n Learning through laughter<\/strong><\/p>\n Refsgaard, who is based in Copenhagen, is a true maverick of machine learning. \u201cI\u2019m interested in the ways we can express ourselves through code,\u201d he explains. \u201cI like to make unconventional connections between inputs and outputs, with the computer serving as a translator \u2014 a tool might allow you to play music with your eyes, or it might generate a love poem from a photo of a burrito.\u201d Refsgaard\u2019s particular spin on innovation isn\u2019t about directly solving problems or launching world-changing startups. Instead, he simply seeks to \u201cpoke at what can be done,\u201d providing accessible open-source templates to prompt new creative ideas and applications.<\/p>\n Programmed by Refsgaard and featuring a custom set of sounds created by Hall, \u201cDoodle Tunes\u201d and \u201cSounds from the Mouth\u201d demonstrate how original compositions can be generated through a mix of spontaneous human gestures and algorithmically produced outputs. In \u201cDoodle Tunes,\u201d a machine learning algorithm is trained on a dataset of drawings of different instruments: a piano, drums, bass guitar, or saxophone. When the user sketches one of these images on a touchscreen, a sound is generated; the more instruments you add, the more complex the composition. \u201cSounds from the Mouth\u201d works through facial tracking and self-capturing images. When the participant faces a webcam and opens their mouth, an autonomous snapshot is created which bounces off the notes of a piano. To try the projects for yourself, scroll to the end of this article.<\/p>\n Libraries, unlimited<\/strong><\/p>\n Saxophone squeals and digital drum beats aren\u2019t the only sounds issuing from the areas where the projects are installed. \u201cMy office is close by,\u201d says Hall. \u201cSo when I suddenly hear laughter, I know exactly what\u2019s up.\u201d This new sonic dimension of the Lewis Music Library fits with the ethos of the environment as a whole \u2014 designed as a campus hub for audio experimentation, the library was never intended to be wholly silent. Refsgaard\u2019s residency exemplifies a new emphasis on progressive programming spearheaded by Boddie, as the strategy of the library shifts toward a focus on digital collections and music technology.<\/p>\n \u201cIn addition to serving as a space for quiet study and access to physical resources, we want the library to be a place where users congregate, collaborate, and explore together,\u201d says Boddie. \u201cThis residency was very successful in that regard. Through the workshops, we were able to connect individuals from across the MIT community and their unique disciplines. We had people from the Sloan School of Management, from the Schwarzman College of Computing, from Music and Theater Arts, all working together, getting messy, creating tools that sometimes worked \u2026 and sometimes didn\u2019t.\u201d<\/p>\n Error and serendipity<\/strong><\/p>\n The integration of error is a key quality of Refgaard\u2019s work. Occasional glitches are part of the artistry, and they also serve to gently undermine the hype around AI; an algorithm is only as good as its dataset, and that set is inflected by human biases and oversights. During a public artist talk, \u201cMachine Learning and the Arts,\u201d audience members were initiated into Refsgaard\u2019s offbeat artistic paradigm, presented with projects such as Booksby.ai<\/a> (an online bookstore for AI-produced sci-fi novels), Is it FUNKY?<\/a> (an attempt to distinguish between \u201cfun\u201d and \u201cboring\u201d images), and Eye Conductor<\/a> (an interface to play music via eye movements and facial gestures). Glitches in the exhibit installations were frankly admitted (it\u2019s true that \u201cDoodle Tunes\u201d occasionally mistakes a drawing of a saxophone for a squirrel), and Refsgaard encouraged audience members to suggest potential improvements.<\/p>\n This open-minded attitude set the tone of the workshops \u201cArt, Algorithms and Artificial Intelligence\u201d and \u201cMachine Learning for Interaction Designers,\u201d intended to be suitable for newcomers as well as curious experts. Refsgaard\u2019s visits to music technology classes explored the ways that human creativity could be amplified by machine learning, and how to navigate the sliding scale between artistic intention and unexpected outcomes. \u201cAs I see it, success is when participants engage with the material and come up with new ideas. The first step of learning is to understand what is being taught \u2014 the next is to apply that understanding in ways that the teacher couldn\u2019t have foreseen.\u201d<\/p>\n Uncertainty and opportunity<\/strong><\/p>\n Refsgaard\u2019s work exemplifies some of the core values and questions central to the evolution of MIT Libraries \u2014 issues of digitization, computation, and open access. By choosing to make his lighthearted demos freely accessible, he renounces ownership of his ideas; a machine learning model might serve as a learning device for a student, and it might equally be monetized by a corporation. For Refsgaard, play is a way of engaging with the ethical implications of emerging technologies, and Hall found himself grappling with these questions in the process of creating the sounds for the two installations. \u201cIf I wrote the sound samples, but someone else arranged them as a composition, then who owns the music? Or does the AI own the music? It\u2019s an incredibly interesting time to be working in music technology; we\u2019re entering into unknown territory.\u201d<\/p>\n For Refsgaard, uncertainty is the secret sauce of his algorithmic artistry. \u201cI like to make things where I\u2019m surprised by the end result,\u201d he says. \u201cI\u2019m seeking that sweet spot between something familiar and something unexpected.\u201d As he explains, too much surprise simply amounts to noise, but there\u2019s something joyful in the possibility that a machine might mistake a saxophone for a squirrel. The task of a creative coder is to continually tune the relationship between human and machine capabilities \u2014 to find and follow the music.<\/p>\n \u201cDoodle Tunes<\/a>\u201d and \u201cSounds from the Mouth<\/a>\u201d are on display in the MIT Lewis Music Library (14E-109) until Dec. 20. Click the links to interact with the projects online.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"