Latent Terrain Synthesis

Welcome

Neural audio codec (autoencoder) is a module used in many AI music generation systems. By unpacking a codec, one can directly interact with the sound generation process, and build tools that link to sensors, hardware, gestural controllers...

Latent terrain is a tool to build corpus-based sound spaces/maps/materials to steer neural audio codecs (such as RAVE, SAME used by Stable Audio 3, Music2Latent). A terrain is a surface map for the codec's latent space, taking coordinates in a control space as inputs, and producing continuous real-time latent vectors that can be used for sound synthesis.

Latent terrain aims to open up the creative possibilities of latent space walk, allowing one to adapt the latent space of a codec to easy-to-navigate interfaces. An example latent space walk with Stable Audio Open 1.0:

Example applications

• Steering a neural audio autoencoder (tutorial coming soon).
• Building 1D/2D latent granular synthesiser (tutorial coming soon).
• Latent looping device.

Supported codecs

Latent terrain can work with any audio autoencoder as long as it offers latent variables. However, only a limited number of them have been implemented for MaxMSP, and we have only tested the following models:

• RAVE
  Realtime Audio Variational autoEncoder for fast and high-quality neural audio synthesis, by Antoine Caillon and Philippe Esling.
• Music2Latent-Scripted
  Music2Latent is a Consistency Autoencoder to encode and decode audio samples, by Marco Pasini, Stefan Lattner, and George Fazekas. We're using a scripted fork of the original repository.
• Stable Audio Open 1.0 (autoencoder)
  [Only supported by nn.terrain oct-2025 version] The pretransform audoencoder in Stable Audio Open 1.0 (not the latent diffusion model, only the audoencoder).
• SAME-S (Semantically-Aligned Music Autoencoder)
  [Only supported by nn.terrain oct-2025 version] The pretransform audoencoder in Stable Audio 3 (not the latent diffusion model, only the audoencoder).

Get started

• Download and Installation
• Instructions
• Pre-Trained Terrains (Presets)
• Compiling from Source

Get in touch

Hi, this is Shuoyang (Jasper). nn.terrain~ is part of my ongoing PhD work on Discovering Musical Affordances in Neural Audio Synthesis, supervised by Anna Xambó Sedó and Nick Bryan-Kinns, and part of the work has been (will be) on putting AI audio generators into the hands of composers/musicians.

Therefore, I would love to have you involved in it - if you have any feedback, a features request, a demo / a device / or anything made with nn.terrain, I would love to hear. If you would like to collaborate on anything, please leave a message in this feedback form.

How To Cite

If you use the software or the resources, we would appreciate citations to the following reference:

Shuoyang Jasper Zheng, Keigo Yoshida, Nico García-Peguinho, Jiatong Liu, Dan Hearn, Anna Xambó Sedó, and Nick Bryan-Kinns. 2026. Latent Terrain: Adapting Neural Audio Autoencoders as Design Materials in NIME. In Proceedings of the International Conference on New Interfaces for Musical Expression. pdf

@inproceedings{zheng_latent_2026,
	address = {London, UK},
	title = {Latent Terrain: Adapting Neural Audio Autoencoders as Design Materials in NIME},
	booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
	author = {Zheng, Shuoyang Jasper and Yoshida, Keigo and García-Peguinho, Nico and Liu, Jiatong and Hearn, Dan and Xambó Sedó, Anna and Bryan-Kinns, Nick},
	year = {2026}
}

Acknowledgements

Shuoyang Zheng, the author of this work, is supported by the UKRI Centre for Doctoral Training in Artificial Intelligence and Music [EP/S022694/1].