My ‘performance technologies’ project is coming along nicely – currently inhabiting a single MAX patch is a program which does the following:
- Search the internet for images given a specific key word
- Analyse the image for colour content and play some corresponding sounds
- Look at a live video feed and find red, yellow, blue and green areas in the video
- Generate a simple sequence given each video feed
Each of these stages is relatively complex – if i have time I’ll post details of how to do each thing on a separate page under ‘MAX’. Here I’ll run through a few general problems I came up against and how I solved them!
Searching the web for images
Using MAX there are a number of strategies for doing this, ranging from the simple (using a built-in object for downloading data) to the complicated (for example, using an external that launches a shell script that does a search). Using the built-in object is relatively straightforward and the process results in a big text file that contains the URL of several hundred images. Decoding this text file and extracting the right bits of data is fiddly, but not impossible (and requires the use of ‘regular expressions’, which are the hardest bit!).
I found the biggest challenge was actually getting the data in the first place. The best way of automatically searching for images seems to be to use flickr – they have a public API which means you can make search requests and get results from some code, rather than using a web interface. A simpler way is to create an RSS feed, within which you can embed a keyword search term. The resulting feed contains a URL for each image within it, which can easily be stripped out and used to download and display the image.
Analysing the image and playing corresponding sounds
I’ve discussed the technical aspects of this in a previous post. Actually the major challenge here is working out what sound goes with each colour – which i suppose is one of the major challenges of the project. Some tentative survey results have started to reveal the answer – i think this might need a dedicated page though!
Finding red, blue, green, yellow in a video feed
For this, as with the static image colour analysis, the first thing i did was reduce the resolution of the image to a 32 x x32 grid – this makes it easier to create musical sequences as the end of the process. The jitter object ‘jit.findbounds’ can be used to extract the specific regions of a given colour – though the source image needs to have it’s brightness, contrast and saturation increased to make the process easier. The jit.findbounds object takes a threshold level for each colour to look for and outputs the relevant coordinates – which i’ve fed into a second matrix for display and analysis. This is mapped onto a live.grid object for sequencing and is further mapped into a more musical form using a pre-defined scale stored in a cool object.
Generating a sequence corresponding to each channel
My original idea was to analyse the content of the individual colour channel all at once – i.e. the whole matrix in one go. I’d hoped i could create a chord from this, perhaps mapping pitch from height of detected colour in the matrix, tone to colour intensity and left / right position to pan. However, i tried various methods for achieving this ‘all in one’ analysis and fell short each time (or created a patch that was so convoluted it wouldn’t actually run).
In the end driving a sequencer from each colour has worked out well. I need to work on the mapping between the sequence and the sound, making sure there is a definite difference between the sound of each channel, perhaps with a distinct set of sounds running from low to high on the grid.