I did the music and sound design for Volatilecycle’s motion graphics showreel…
Here’s a remix i did for I Am Jupiter, from their upcoming EP ‘Trucks’:
For a while now I’ve known i wanted to be able to use data to generate interesting music procedurally, or algorithmically. Although not a new idea, i like the possibility that the control for the algorithms could come from an image- or some other realtime data, actually – and that the music would be related somehow to the data but would be a piece in it’s own right.
Although there are lots of ways to think about generative music – formal grammars, Markov chains, state machines, and so on – i’ve not really been sure how my software would work.
It doesn’t take a genius to associate generative music with Brian Eno – aside from his various ambient albums, he’s known for various generative programs such as Koan and Bloom, and for developing some generative music for the generative computer game ‘Spore’. His most recent app, called Scape and developed by Eno and Peter Chilvers, adopts an approach whereby little graphical modules are dropped onto the screen, each one having a simple set of generative rules. The interesting part is that each module kind of knows what the other modules are doing, so they interact to produce more complex, interesting music. That got me thinking about my own project, and how i could modularise the generative ‘engine’, creating simple building blocks that could work together. For example, there could be a module who’s job would just be to play a long note every 1 – 3 bars, with rules to govern how often notes are played, what the notes are, what the tonal quality of the sound is and so on. Since the whole thing will be governed by incoming data, there will be ‘global’ parameters that all the modules can use – but equally, different modules could be made aware of each other and interact somehow.
A basic sketch of the system at the moment looks like this:
So all the modules are linked, and there’s a global arrangement generator and key / chord progression generator, as well as a module for a range of different sounds, each one with it’s own set of rules for making stuff happen.
So far i’ve only got the pad generator… but i like the idea that each module could have a little personality too. Maybe the pad generator’s a bit lazy and laid back, and the lead melody generator is highly strung and liable to have a strop if it doesn’t get it’s own way!
I’ve spent an awful long time working out seemingly trivial technical points with this project so far, but now i feel like I’m reasonably fluent in using Puredata and Openframeworks, and have created a reasonably flexible development setup using oF, Pd and Logic Pro as a temporary sound source. Although the ultimate goal is to integrate everything into a single application, having the component parts separated out like this means that i can experiment and make changes, and see the results quickly.
The use of the ‘vanilla’ Pd version means that it should be possible to integrate it into the OF project using the ‘OfxPd’ add-on. I’ve already done some experimentation with this, and simple Pd patches are easy enough to integrate. The sound generation will be taken care of inside Pd eventually, rather than using Logic – so the whole thing should work as a single app.
The OF program loads an image and steps through it from left to right – taking a cue from how a book or musical score is read. Each pixel column is analysed, and a measure of the average values in the column calculated. Clearly this doesn’t tell the whole story of the image – variations from top-to-bottom are not taken into account yet, for example – but it’s enough to go on for now.
So for each column i’ve got an average measure of hue, brightness and saturation for that column, a kind of running average for each one as well, and a total H/S/B value per column. The running average is used to work out how much the current column’s average changes for each step – this is used to provide a kind of ‘complexity index’ as a measure of the amount of variation across the image.
The image is read left-to-right at a rate of one column per 16th beat, at (approx) 100 BPM. For a reasonably wide image (say, 1024 pixels) this gives a total song length of around 64 bars, or 2.5 minutes.
All this data is sent via MIDI (which turns out to still be the best way!) to Pd, which, at this stage, generates some simple melodies using a basic pad sound created in Logic. The pad generator is governed by some simple rules, such as:
More complexity = more variation, fewer spaces between notes
More brightness = more likelihood of a major key progression over a minor key progression
‘Red’ spectrum = warm, filtered sounds.
‘Blue’ spectrum = cold, bright sounds
Brightness is also literally mapped to filter cutoff, meaning the brighter an image is, the brighter a sound is.
Results so far
Feeding the program different images certainly results in the generation of short pieces of music that differ in terms of tonal quality, note progression and ‘feel’. The music so far is pretty boring, but i think that can be addressed by introducing more interesting generative rules. I think as a ‘proof of concept’ what i’ve got so far is positive – definitely darker images produce darker sounds, different colours produce different results and transitions between contrasting parts of an image produce corresponding changes in the audio.
Here’s a couple of examples:
I’ve made good progress in understanding what i can do with OpenFrameworks, how i can get it to interact with PureData and what the limitations of it are. Learning OpenFrameworks has proved challenging – although there are plenty of ‘getting started’ tutorials which make the basics look easy, there’s a real jump up to the more complicated stuff that requires a good depth of understanding of the world of C++.
Some of the real sticking points so far have been in getting OF to work with a range of external ‘add-ons’. I feel like i’ve wasted too much time trying to deal with fiddly programming issues, and not enough time prototyping and trying things out.
The setup i’ve got (more or less) working to try things out, without relying on my shaky programming knowledge too much, is as follows:
OpenFrameworks loads an image and performs some analysis, possibly calculations based on the image. Data is passed over the local network (or MIDI) to PureData which creates the musical data. Sounds are generated using a standalone software instrument (NI Kontakt at the moment).
Eventually everything will be contained inside a single OF application, and then ported to iOS…
The next step is to start experimenting with sound generation, working with a set of rules like these:
- colour content relates to tonal quality – more reds = warm sounds, more blues = cold sounds
- average brightness relates to key – brighter = major, duller = minor
- Melodic content will be derived from an algorithm – i’m going to try something simple to start with then something more complicated, like utilising John Conway’s Game of Life to generate a Markov chain governing the probability of a note being generated.
- Rate of change of a parameter (like hue, or saturation) will govern complexity of melodic content.
I’ll use a standard major / minor key, with weighting to favour the ‘easier’ notes.
Hopefully i should be able to post some examples soon.
I’m making slow and steady technical progress through the project – i’ve started to write simple applications based on OpenFrameworks using xcode. I’m going to do the graphical stuff with this, ‘wrapping’ a Puredata patch into the application. PD will deal with the sequencing and, ultimately, the sound generation too.
Here’s the results of an OF application that analyses an image and graphs the average hue, saturation and brightness for each column of pixels:
Here’s a prototype PD patch that generates sequences who’s complexity (as in, the number and variation of notes in the sequence) can be controlled in a probabilistic way – the variations in image data as seen above will be ‘mapped’ to this sequence generator.
When this prototype is complete, it should be able to generate musical sequences that are unique to each image presented. If this method for mapping image data to music works I’ll develop it further, adding more sounds and more sophisticated sequence generation processes. It should then theoretically be possibly to substitute the image data with something else too.
Posting some pictures from last year’s ‘colourorgan’ project – here a yellow chair is offered up, the software presenting a corresponding yellow ‘matrix’ on screen (that was used to drive an audio sequence) and an image tagged as ‘yellow’ from a Google image search:
Here’s a couple from the IOCT showcase – the software in action again:
And a rather fetching picture of me explaining something unwittingly sporting a large image of a skull:
My original proposal for my Master’s final project went along the lines of:
“Exploring the world through music: an investigation of data sonification (and related techniques) in computer generated music and audio mixing.”
I’d done a bit of work in the past around the idea that you could analyse an image and have some sort of music generated. The ‘colourorgan’ used colour information to generate / play specific sounds and sequences, and the ‘Flickr Playr’ took a Flickr stream and made ambient music from the images in it. For this project i want to explore more ideas around audification, sonification, and generating music from various data.
It’s been a while since i’ve worked in earnest on the project, but i’ve recently started revisiting some of the original ideas and re-working elements of the Flickr Playr. I’ve also come up with a slightly snappier title for the project – ‘Otono’, which sort of translates to ‘Sonic’, or ‘Tone’, from Japanese.
Quite a lot of the research carried out since the proposal has been around the various technologies that I could use to complete the project. I’m keen to keep things open-source where possible, and have moved away from using MAX/MSP in favour of Puredata. I’ve also been looking at OpenFrameworks for developing graphical, user interface and analytical tools, and, since i’m still working on Mac OS, using Xcode to develop the project with a view to creating a mobile app as well.
I’m going to try and post interesting developments to this blog – watch this space!
Last year i made a patcher called the ‘Flickr Playr’ that looks at your Flickr photo stream, downloads the images and creates ambient music based on the image. I’ve finally fixed all the problems with it and created a nice simple interface - you can enter your flickr ID and generate some nice music! Here’s a little video demo. When i get round to it i’ll post the app here too!
The date for the IOCT showcase looms – i have just over a week to decide exactly what i want to put on, and prepare the practical stuff (as well as doing the writeup for the colour organ piece).
My original plan was to add a control system to the organ, building a reactable style control table. While i still want to build this, given the time constraints i don’t think it’s practical. Also i’m wondering at the relevance of the technology, and given the potentially home-made nature of the product its reliability. Had i focussed on this for the performance technologies project i might have been able to spend more time testing it and be a bit more positive about it. I’m not ruling it out at this stage but i’ll see how i feel after i’ve done a bit more work on other stuff!
My main focus will be adapting the ‘colour organ’, breaking it into smaller parts which still incorporate some participation but will hold up reliably in the more chaotic environment of the showcase. Currently i’m thinking the web search / musical analysis part will focus on a specific flickr group, and produce music from provided images which is much less random – so, for example, the same image will produce roughly the same piece of music each time. I could have the feed live-updated with images from the showcase.
The live camera input(s) will focus more on movement detection, rather than colours – I have some pre-recorded video clips that i can use to demo this, and may have a couple of webcams about the place which detect, say, people entering the building, and play some music accordingly. I might also feature a lava lamp here – yep.
I could also do a ‘take your photo, see what you sound like’ booth thing – perhaps subtracting the background image and focussing on the face / clothes of the person. I’ll see how much i get done!
I’m going to build everything around this website, and include some audio works too. So i’ll need a means of playing music from soundcloud as well – perhaps an ipad / headphones, maybe a laptop / iMac.
I need to think about the kit I’ve already got, and how to use it – i don’t want to ‘waste’ the projector i found and not use it.