Jazz Hands

Overview

A web-based participatory musical experiment created with Eric Rabinowitz for our Connections Lab class at NYU. The concept: turn everyone’s phones into instruments in a conducted choir, inspired by Jacob Collier’s audience choir performances. People tilt their phones up and down to change pitch within the current chord, shake for vibrato—movements that look like conducting or, well, jazz hands.

The idea came from both of us having done participatory music projects before and wanting to see what we could build together quickly. We settled on the phone as the object of attention—ubiquitous, sensor-rich, intimate.

How It Was Made

Built entirely for the web using WebSockets for real-time synchronization and Three.js to give each person a virtual hand in shared space. The interaction model:

• Gyroscope: Tilting the phone up/down switches between pitches in the current chord
• Accelerometer: Shaking introduces vibrato
• Score: Pre-composed blocks of jazzy chords that the conductor (us) triggered

Each phone became a voice in the choir, with the physical gesture of tilting and shaking translating to musical parameters. The Three.js hands gave visual feedback—you could see your contribution and everyone else’s in virtual space.

Technical Approach

WebSockets handled the real-time communication between all the phones and the server. Three.js rendered the spatial representation of participants’ hands. The musical logic mapped sensor data to pitch selection within chord voicings—keeping everything harmonically coherent while giving people expressive control through gesture.

The “jazz hands” name came from the very jazzy chord progressions we composed and the literal hand-waving movements people made while playing.

Context

Class project for Connections Lab, NYU IMA (2024). A quick exploration of phone-as-instrument and collective music-making that drew on both our backgrounds in participatory performance and interaction design.