After connecting my Kinect to my computer I really wanted to play around with some hand tracking algorithms. I wrote a basic processing brightness tracking code that sort of worked but I knew that using some other libraries could really improve my results.
With some Googling I managed to find a library canned TUIO. “TUIO is an open framework that defines a common protocol and API for tangible multitouch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and tangible object states.”
After a few hours of playing around with this software/library pack I was amazed on how powerful it really is. They have just released an open source app for OS X called TuioKinect. This program controls your Kinect and outputs multiple points. By setting the forward and backward threshold I was able use this to detect both my hands and output the X and Y coordinates.
Since this software was not just designed for Kinect, there are already 10s of different clients for different codes already written and optimized. I took a loot at the first example using Actionscript but could not manage to get it to work. There was some UDP socket that you are required to run to patch the connection from TUIO to Flash. I did manage to get the processing example working and the results were better then I expected. Here is a screenshot of the results. The lines on the right are the tracked movements from the hands on the left.
Setting up this does require a few seconds and is not optimal for end users. There must be a way to calibrate the front and back thresholds without anything input.
Clearly the next step that is required is a full body skeleton tracking algorithm which can use vectors for each joint just like the calibration in the xbox menu is.
If you would like to download TuioKinect visit: https://code.google.com/p/tuiokinect/
I have also found a really good example of a fluid library for processing that is already integrated with TUIO. MSAFluid is a library for solving real-time fluid dynamics simulations based on Navier-Stokes equations and provides some amazing visual effects. Here is a picture of its output with the Kinect.
*Update* If anyone can not find the link to the MSAFluid example it is here: http://memo.tv/files/memotv/MSAFluid.zip
To run this demo just run TuioKinect first, then run “MSAFluidTuioDemo” from processing and start moving your hands.
I think im going to be playing around with this app for a while! Like always, if you have any questions or comments please leave them bellow.