Improvisational Syntax

Overview

A Node.js program that freely improvises music while learning from and reacting to a human performer in real-time. Built for my NYU Tisch IMA thesis, this virtual improviser explores what happens when you teach a machine to speak your exact musical language—then hide its decision-making process to restore the human element of surprise.

The project stems from a decade of studying jazz and free improvisation, combined with influence from George Lewis’s pioneering work on virtual improvisers and the live coding scene’s philosophy of radical transparency. Rather than using machine learning or generative AI, the system uses layered intervallic decision-making that mimics how I personally make choices while improvising—creating genuine dialogue instead of predetermined responses.

How It Was Made

The virtual improviser runs on Node.js using Tonal.js for music theory operations and WEBMIDI.js for real-time communication, with Ableton handling audio output. I deliberately avoided typical computer music tools like Max MSP or Pure Data, instead building in JavaScript—a language I’m comfortable with and have used for several music projects.

The architecture uses a layered system where each layer influences how the program plays its instrument. Minor, short-term adjustments might slightly shift pitch or increase rhythmic density, while broader long-term modifications affect overall phrasing by changing dissonance levels or harmonic direction. This layered approach models the actual decision-making process of improvisation, but specifically reflects my personal improvising routines and vocabulary.

The technical challenge wasn’t building something that could generate music—it was building something that could improvise the way I do, establishing enough mutual understanding that we could genuinely co-create.

Technical Approach

No machine learning. No black boxes by default. The entire decision-making process could theoretically be made transparent in real-time—showing exactly why the program chose each note, revealing its analytical process as it happens.

But here’s the twist: after building all that radical transparency into the system, I deliberately hide it during performance. The program reads and responds to musical information, but I don’t see its decision-making process. This creates the same kind of imbalance humans navigate when improvising together—we analyze body language, eye contact, room atmosphere, all that non-verbal communication. By hiding the machine’s “thought process,” I restore balance. Neither of us gets perfect information about the other’s next move.

The virtual improviser has learned my language. It knows my vocabulary, my tendencies, my routines. This mutual understanding—this trust—makes genuine co-creation possible even without seeing inside each other’s heads.

Context

Created as my thesis project for NYU Tisch’s IMA Low Res program (New York, Berlin, Shanghai, 2023-2024). The project draws from George Lewis’s work on virtual improvisers like Voyager, the live coding community’s approach to transparent human-computer collaboration, and jazz tradition’s emphasis on dialogue and co-creation.

The work argues that radical transparency—showing all decision-making processes—can enrich improvisation for all participants. But it also explores the flip side: sometimes hiding that transparency creates more human, more organic interaction. The black box approach can feel less mechanical precisely because it maintains surprise and spontaneity, much like we don’t fully understand another person’s thought process when we play together.