kinect plugIn Map Skeleton Point To Depth


Before going any farther here is what I’m attempting to do. Using the MS kinect plug, I want do display the player’s silhouette on a plane. I want to simulate that this silhouette is holding two game objects

I am looking for an equivalent function to MapSkeletonPointToDepth but could not find it (probably didn’t look in the right file).

I’ve also constructed a projection matrix based on the kinect sensor specs and then project the joint position returned by the kinect by this matrix and then I’m doing some math to calculate where that point will fall on the plane’s texture and then more math to estimate where’s the 3D position on the plane that I have to move the game objects so they look like they are being hold by the player’s silhouette.

So back ti my question, is there an equivalent function to the MapSkeletonPointToDepth provided by the kinect SDK? or do I have to compile an kinect dll/plug in?

Thanks in advance

I’m posting what I have so far. I have to mention that I have not tried the dll method as I do not have Unity Pro. The other method I’m using, I have been able to do what I want to do but the scale is completely off, so I’m trying to figure out if I can just have and offset value to fix this or is I can change any of the calculations

If you have Unity Pro, all you have to do is add to the kinect plug in access to the function “NuiTransformSkeletonToDepthImage” of the Kinect10.dll. You need pro as Kinect10.dll is an unmanage dll. I haven’t try it yet but according to MS it should return the pixel position of that joint on the depth map image. This should be similar to the method “MapSkeletonPointToDepth” in the Kinect SDK.

As I am not buying Pro yet, here is the other solution a came up with. It involves some math to calculate where in the texture is the joint. Here are the steps:

  1. Construct a projection matrix using the specs of the kinect
  2. Multiply the joint position with this projection matrix and the X and Y values should be in a range of -1 to +1 and -AspectRatio to +AspectRatio. I found a simpler way of doing this without much math which I’m describing at the end and skipping the next steps
  3. Normalize those values to a range of -1 to +1. This way the value will indicate where the joint is with respect to the center of the plane as a percentage value (%) so if x = -0.5 then pixel x is half way between the center of the plane and the bottom of the plane (you get the idea)

If you are using Unity Pro and add access the the dll function. You will need to convert the pixel value into a 0 to 1 value and then convert that to a range of -1 to +1 by doing (x - 0.5) * 2. You do the opposite operations that are used when unpacking normals

Once you know where the joint in pixels coordinates is base on the -1 to +1 range, you can calculate where that joint is on 3D space base on the value.

In order to not calculate the projection matrix, we have to use two different isosceles triangles which bases go from -1 to +1 and the unique angle corresponds to either the vertical (alpha) or horizontal (beta) kinect field of view angle. From this we can deduct:

  1. For the horizontal angle, we can calculate that the distance from the kinect to the base of the triangle is cotangent(alpha / 2) and for the vertical cotangent(beta / 2)
  2. With some trig we can get the relations of : cotangent(alpha / 2) / pixel_X = z / x and cotangent(beta / 2) / pixel_Y = z / y. Solving those equation for pixel_X and pixel_Y will return values between -1 and +1

Regardless what method you use to get the xy values between -1 and +1, here is how I calculated where it will land on 3D:

  1. positionX = localScale.x * 5 * 0.5 * pixel_X
  2. positionY = localScale.y * 5 * 0.5 * pixel_Y
  3. objectToControl.position = (new Vector3(positionX, positionY, offSet)) + plane.position

I multiply by 5 as the unity plane has 10x10 dimensions and by 0.5 in order to use half the scale.
OffSet is how far from the plane on the z-direction the obejct will be.

If the math/logic is hard to understand, let me know and I’ll make some drawings to make it easier to understand