Use the Up, Down, Left, and Right arrows to navigate the space. Shift + Up, Down, Left, Right changes your viewing angle. R resets. (This may take a long time to load! Sorry, working on that...)
The [dot] MP5 project is part of my ongoing exploration of new media platforms for music and visual artists. MP5 is an invented term that relates to the two famous file formats: the MP3 nad MP4. In the MP3, sound (audio) is digitally encoded in one file. In the MP4, sound and graphics (audiovisual)... In the MP5, sound, graphics, AND interaction (interactive audiovisual = audiovisualactive ?). The [dot] in the project title represents the possibility of a time when interactive audiovisual media becomes prevalent enough to develop standard file formats for such media, like the MP5.
In the MP5 constructed here, I set out to create an interactive audiovisual landscape representing the music and aesthetic lens I have been constructing for my next EP. The experience features a few different tunes and musical ideas in progress, mostly focused on groove, texture, and lyricism. The tunes are paired with shapes, images, colors, and animations, and placed carefully in the 3D space so as to create a narrative. Mostly the user moves forward through the space, but they can diverge sideways or look around in any direction. The space invites you to explore. As you move around, sounds and shapes pan around you depending on their relative location. I've found it useful to think of these sounds and shapes simultaneously as soundshapes. Composing with soundshapes is a unique experience. Thinking of sound, visual aesthetics, and 3D geometry all at once can be taxing, but rewarding all the same. New possibilities open up when you can breathe life in to one form of media by animating it with another.
The MP5 shares a lot in common with video games. Unlike most video games, the MP5 is not goal-based. It is meant to be experienced actively, but without haste or concrete objective. Like taking a walk to hear the birds, or running while listening to music, the MP5 is more of an open experience. Like its theoretical predecessors, the MP3 and MP4, you press play and engage as you wish.
I think it would be great if there was something like a DAW in which soundshapes could be composed and animated in 3D space. Like Logic, REAPER, or Ableton, but opening up that visual dimension. Tons of tools have been developed to do something like this: Unity, Processing, openFrameworks, p5js. But few are tailored for musical or visual artists. What if there were tools that made it easier to piece something like this together? All you need to do is import media of your choice, create shapes, drag them around a space, apply audio fx as desired, record on tracks that are linked with shapes or positionsin the 3D space, etc... Maybe this is simply a question of how/when tools like Unity become easy for more people to use.
I constructed this MP5 project in the browser, using p5js in conjunction with the WebAudio API. I'm using p5js in WEBGL mode for 3D graphics. I am interested in doing all of this in the browser primarily for wider access. It's important to me that nearly anybody could access my work. I want my work to be relatable and speak whatever meaning it can for whoever needs it. While I have not fine-tuned this particular project yet, this technology stack allows anyone to access the project on the web, whether on a mobile device or desktop computer.
Each soundshape in the work has (1) a variable form, position, and size, (2) a particular sound in the form of one or more Web Audio nodes (sample or synthesis), and (3) is linked to a spatial HRTF panner node and audio analyser node. The analyser node allows animation of the shape based on the sound it's making... imagine shapes pulsating or changing color with the volume of the sound.
I constructed most of the sounds for the tunes in Logic. I bounced each stem and loaded the samples individually (using the AudioBuffer and AudioBufferSourceNode from Web Audio) so I could spatialize the audio in the browser if desired. I also constructed some sounds in Max MSP, and performed some basic audio synthesis in the browser.
For the visuals, it took a while to get used to WEBGL and I still have a lot to learn. Mostly, I use basic geometries, apply images and videos as textures on their surfaces, and animate the shapes based on the sounds they are linked to. I created the videos and images using my iPhone camera and crafty combinations of Preview (mostly the Instant Alpha feature to remove colors) and Paintbrush (essentially MS Paint on Mac). I like when images have a collection of pixels with zero alpha so that they are partially transparent. Applied to 3D geometries as textures, the resulting shapes look otherworldly and when stacked one behind the other produce interesting visual artifacts in the transparent regions. In general, I'm very interested in the visual artifacts that result from moving through these shapes, seeing their textures applied from the inside, and the artwork that emerges from glitchiness in navigating this MP5 space.
If you have any questions about how I put this project together, feel free to reach out (mulshine@stanford.edu). Otherwise, enjoy!
