Teaching AI to Work Across Symbolic Notation and Audio
A best paper-winning model from NUS Computing helps AI learn from both written notation and recorded audio within a single framework, expanding how machines can understand music
A page of sheet music and a recording of the same piece carry different kinds of information. The score tells you the architecture, while the recording tells you what a particular performance did with it: a held breath before the chorus, a guitar slightly out of tune in a way that somehow works. A trained musician can hold both at once without thinking about it.
Most AI systems built to understand music have had to pick one. Notation formats like ABC or MIDI give a model precise access to structure but nothing of how a piece actually sounds. Waveform audio preserves acoustic detail, but does not explicitly mark notes, bars, or other symbolic structures. Building something that could work across both turned out to be harder than stapling two existing tools together.
The paper, “Integrating Symbolic and Waveform Music into Large Language Models,” was authored by PhD candidates Tu Teng and Liu Xiaohao of NUS School of Computing, Assistant Professor Ma Yunshan of Singapore Management University, Postdoctoral Fellow Qi Ji of Tsinghua University, and KITHCT Chair Professor Chua Tat Seng of NUS School of Computing. Their model, UniMuLM, was awarded Best Paper at the 32nd International Conference on Multimedia Modeling (MMM 2026), held in Prague in January 2026. It is among the first frameworks to give a large language model symbolic notation and waveform audio at the same time, and have it make sense of both together.
A Timing Problem, Mostly
Plenty of good encoders already exist for waveform audio, and plenty of good tokenisers exist for symbolic notation, developed over years of separate research tracks. What had kept them from working together was more basic: the two formats measure time differently. Symbolic music organises itself into notes, bars, and phrases, each a defined unit with a clear boundary. A waveform has none of that. It is a continuous stream of samples, sampled far more densely than a score is written, with no built-in markers for where one musical idea ends and another begins. Line the two up using a fixed clock and the detail starts to blur: a waveform’s texture flattened out, or a score’s structure lost in translation.
UniMuLM works around this with a unified encoder that reads both formats through three layers of musical structure: beat, bar, and phrase. Each granularity builds on the finer one below it, allowing the encoder to combine local and longer-range context. An alignment module, trained with a contrastive loss and a cross-reconstruction objective, pulls the symbolic and waveform representations into a shared space, so that a bar of notation and the seconds of audio it corresponds to end up encoded as related information rather than separate inputs.
Building on Llama3-8B, the team trained the model in three stages: first teaching the encoder to align the two music formats with each other, then warming up the language model with music theory knowledge from symbolic datasets, and finally fine-tuning across tasks that mix notation, audio, and written instructions together.
What the Results Showed
The team evaluated UniMuLM across four music understanding benchmarks covering music description, question answering, theory knowledge, and symbolic melody tasks.
Rather than focusing on a single capability, the evaluation examined whether a single framework could support different types of musical information processing.
According to the paper’s benchmarks, UniMuLM was competitive with several waveform-capable and audio-language baselines, with GPT-4o included as a reference. On a benchmark testing music theory knowledge, it scored 0.460, ahead of GPT-4o’s 0.441 and ChatMusician’s 0.354. It also did better than both at continuing an unfinished melody and filling in a missing section of a tune.
The team also tested what happens when parts of the training process are stripped away. Removing components of the alignment process made it harder for the model to connect symbolic notation with audio, while skipping the music-knowledge training stage also reduced its performance on several downstream tasks.
Where This Could Go
Music AI tools have tended to be good at one job: reading a score, or listening to a recording. A model that can hold both could support a wider range of work. A transcription tool that turns a recorded performance into notation. A composition assistant that reasons about a piece’s harmonic structure while also judging how a rendered version of it actually sounds. A captioning or critique tool that draws on both what is written and what is heard.
The researchers are upfront about the limits of what they have built so far. The alignment training used single-track, single-instrument recordings within Western tonal and metrical conventions, so it is untested on other musical traditions or looser rhythmic structures. Because the model generates output through a text-based language model, its outputs are currently limited to text and notation rather than MIDI files or waveform audio. The team names a hybrid decoder, one able to generate both symbolic and acoustic tokens, as a direction for future work.
Tu, T., Liu, X., Ma, Y., Qi, J. and Chua, T.S. (2026) “Integrating Symbolic and Waveform Music into Large Language Models,” In 32nd International Conference on Multimedia Modeling (MMM 2026), January 29–31, Prague, Czech Republic, https://doi.org/10.1007/978-981-95-6957-1_7
