Exploring Gestural Affordances in Audio Latent Space Navigation
Author: Shuoyang Jasper Zheng, Anna Xambó Sedó and Nick Bryan-Kinns
Abstract
The latent space of generative AI models affords unique creative possibilities in the domain of creative AI for music and sound. While interface designs for latent space navigation typically rely on sound-producing gestures that involve bodily motions and movements, the underlying subjective perceptions of these gestures remain underexplored. To understand how musicians perceive sound-producing gestures and tailor them into performance techniques with audio latent space navigation, we present a workshop study of musicians’ embodied exploration of a stylus and tablet interface embedded with an adapted latent space of a neural audio synthesis model.
All supplementary material for the manuscript can be found here:
Instrument Design
-
Generating Latent Terrains
We designed a digital musical instrument with a stylus and tablet interface as a research probe. The instrument embeds Latent Terrain, an adapted form of latent space of a neural audio synthesis model, inspired by wave terrain synthesis. One can navigate the latent terrain using gestures afforded by the stylus and tablet...
Data Collections
-
Graphical User Interface
Each workshop activity has a Graphical User Interface (GUI) page displaying relevant information, instructions, and the instrument’s user interface. The investigator’s laptop controls the switching between GUI pages and switching between terrains in participants’ instrument...
Results
-
Performance Scores
Participants were asked to explore and find pieces of sound or music in each terrain, and then draw a “score” on the canvas to help them perform these pieces. They were encouraged to be creative and use any kinds of notation (graphical/textual) in the score... -
Narrative Analysis
We compiled participants' exploration of gestures and their development of techniques into longitudinal descriptions. This was inspired by previous works in HCI and artist-led research on DMI design to interpret observational data from mixed methods in a systematical way centring around a research question...