[unity windows] build a kinect plugin : freeze on call to NuiImageStreamGetNextFrame

Unity Engine
1

Hi,

I m workin on a kinect plugin for unity 3D 3.5 and I m stuck with the video data

I successfully init the kinect like this and all is ok with skeleton data

bool KinectManager::InitNuiSensor()
{
    INuiSensor * pNuiSensor;

	status = 0;

    int iSensorCount = 0;
    HRESULT hr = NuiGetSensorCount(&iSensorCount);
    if (FAILED(hr))
    {
        return false;
    }

    // Look at each Kinect sensor
    for (int i = 0; i < iSensorCount; ++i)
    {
        // Create the sensor so we can check status, if we can't create it, move on to the next
        hr = NuiCreateSensorByIndex(i, &pNuiSensor);
        if (FAILED(hr))
        {
            continue;
        }

        // Get the status of the sensor, and if connected, then we can initialize it
        hr = pNuiSensor->NuiStatus();
        if (S_OK == hr)
        {
            m_pNuiSensor = pNuiSensor;
            break;
        }

        // This sensor wasn't OK, so release it since we're not using it
        pNuiSensor->Release();
    }

    if (NULL != m_pNuiSensor)
    {
        // Initialize the Kinect and specify that we'll be using skeleton
        hr = m_pNuiSensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_SKELETON | NUI_INITIALIZE_FLAG_USES_COLOR); 


        if (SUCCEEDED(hr))
        {
            // Create an event that will be signaled when skeleton data is available
            m_hNextSkeletonEvent = CreateEventW(NULL, TRUE, FALSE, NULL);

            // Open a skeleton stream to receive skeleton data
            hr = m_pNuiSensor->NuiSkeletonTrackingEnable(m_hNextSkeletonEvent, 0); 


			
			nextColorFrameEvent = CreateEvent(NULL, TRUE, FALSE, NULL);

			hr = m_pNuiSensor->NuiImageStreamOpen(NUI_IMAGE_TYPE_COLOR, NUI_IMAGE_RESOLUTION_640x480, 0, 2, nextColorFrameEvent, &videoStreamHandle);

			if (FAILED(hr))
			{
				  return false;
			}
        }
    }

    if (NULL == m_pNuiSensor || FAILED(hr))
    {
      return false;
    }

	status = 1;

    return true;

}

I want to get a video frame and upload it as a texture in unity

In order to get this done, I ve build a GetPicture method

long KinectManager::GetPicture(unsigned char **buffer)
{
	HRESULT hr;

	NUI_IMAGE_FRAME imageFrame;

	 // Attempt to get the color frame
        hr = m_pNuiSensor->NuiImageStreamGetNextFrame(nextColorFrameEvent, 0, &imageFrame);

      
	if( FAILED( hr ) )
	{
             return hr;
	}

	INuiFrameTexture * pTexture = imageFrame.pFrameTexture;

    NUI_LOCKED_RECT LockedRect;

	pTexture->LockRect( 0, &LockedRect, NULL, 0 );

    if( LockedRect.Pitch != 0 )
    {

			unsigned char *pBuffer = (unsigned char*) LockedRect.pBits;

			NUI_IMAGE_RESOLUTION resolution = imageFrame.eResolution;
        
			if(resolution == NUI_IMAGE_RESOLUTION_640x480)
			{
				*buffer = pBuffer; 
			}
      }



	  return S_OK;
}

the problem is that unity freeze on the call to NuiImageStreamGetNextFrame.

If I remove the line nextColorFrameEvent = CreateEvent(NULL, TRUE, FALSE, NULL); in init. I dont get the freeze any more, but buffer remains empty.

Any suggestions?

2

I m stuck with this… any ideas guys ? thanks !

3

I finally got it to work !

I was missing the UnlockRect(0) line of code on the C++ size which was the root cause of the freeze

anyway, now I ve got problems on the unity size.

I get a out of memory error after about a minute of run.

looks like when I replace image in texture2d I get a big leak.

How to tell unity to delete the previous instance of the texture

IntPtr  ptrImageBuffer = IntPtr.Zero;

				int result = GetPicture(ref ptrImageBuffer);
	
				if (ptrImageBuffer != IntPtr.Zero )
				{
					
					if (this.textCam != null)
					{
						this.textCam =null;
					}
					
					this.textCam = new Texture2D(640,480);
					
					for (int x=0;x<640;x++)
					{
						for (int y=0;y<480;y++)
						{
							
							int posbrush = x*4 + ((480-y) * 640 *4);
					
							int val = Marshal.ReadInt32(ptrImageBuffer, posbrush);
		
							int red   = (val  0x00ff0000) >> 16;
							int green = (val  0x0000ff00) >> 8;
							int blue  = (val  0x000000ff);
		
							Color c = new Color( (float)(red / 255.0f),(float)(green / 255.0f), (float)(blue  / 255.0f) ,1.0f);
							
							textCam.SetPixel(x,y,c);
							
							
						}
						
					}
					
					textCam.Apply();