[microsound] image sonification techniques

[Dara Shayda]

http://orazon-art.com/ressources/tool/mocap/mocap_01.jpg   This is what I was referring to in terms of makeup. You place some sticky dots on your face and then videotape yourself talking or singing and so on.

You can write a little tool (or borrow from some 100s of others who do facial gesture capture research) and extract the 2D/ 3D movements of the face.  

Then redo the video without the dots.

Use the dots as input parameters into you software, so they will serve like a sequencer for local synthetizers you can attach to each dot. Synthesize individual dots and then superimpose the sounds. 

The redone video without the dots can be image synthed frame by frame. You can then have the dots data to drive changes to the audio track of the voice.

So you get three sets of sounds generated from facial gestures.

Similar concept can work with the rest of the limbs. 

There are expensive 3D gear for the whole body and face. But you might be able to do with the sticky dots for the beginning.

D
On 2012-01-10, at 2:03 AM, Kovács Balázs wrote:

> Hi Dara,
> 
> sorry for the late response, and many thanks for the support!
> 
> i plan to begin the interaction-based works in the last part of the project, probably
> the next week? i built a basic eye/mouth follower (http://vimeo.com/31281227),
> but i don't have experiences with phoneme recognition... anyway it would be very
> good to include it in the series, so if You have idea, then welcome and i would join
> in! maybe i could also develop a system for the end of the next weak, but as i see
> you have experiences in the field with this wonderful "Beginning" work!
> 
> All the best,
> Balázs

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://or8.net/pipermail/microsound/attachments/20120110/926e2e97/attachment.html>