Convert Texture2d to opencvsharp to Mat (and vice versa)

Unity Engine
1

I’m using opencvsharp with Unity to detect edges from a webcam, the process is like this:

  • converting the current frame to OpenCVSharp Mat object

  • processing the image (using the canny edge detector)

  • converting the OpenCVSharp Mat of canny image to Texture2D

I have tried to follow the conversion between OpenCVSharp image format (Mat) toUnity3D image holder (Texture2D) from here but the unity editor keeps crashing every time. This is my first project using Opencv (and opencvsharp) and any help would be much appreciated.

For reference here is the code for the convertion that I used:

TextureToMat

void TextureToMat()
{
   // Color32 array : r, g, b, a
   Color32[] c = _webcamTexture.GetPixels32();

   // Parallel for loop
   // convert Color32 object to Vec3b object
   // Vec3b is the representation of pixel for Mat
   Parallel.For(0, imHeight, i =>
   {
       for (var j = 0; j < imWidth; j++)
       {
           var col = c[j + i * imWidth];
           var vec3 = new Vec3b
           {
               Item0 = col.b,
               Item1 = col.g,
               Item2 = col.r
           };
           // set pixel to an array
           videoSourceImageData[j + i * imWidth] = vec3;
       }
   });
   // assign the Vec3b array to Mat
   videoSourceImage.SetArray(0, 0, videoSourceImageData);
}

MatToTexture

void MatToTexture()
{
   // cannyImageData is byte array, because canny image is grayscale
   cannyImage.GetArray(0, 0, cannyImageData);
   // create Color32 array that can be assigned to Texture2D directly
   Color32[] c = new Color32[imHeight * imWidth];

   // parallel for loop
   Parallel.For(0, imHeight, i =>
   {
       for (var j = 0; j < imWidth; j++)
       {
           byte vec = cannyImageData[j + i * imWidth];
           var color32 = new Color32
           {
               r = vec,
               g = vec,
               b = vec,
               a = 0
           };
           c[j + i * imWidth] = color32;
       }
   });

   processedTexture.SetPixels32(c);
   // to update the texture, OpenGL manner
   processedTexture.Apply();
}
2

Hi were you able to solve your problem? I am doing something similar. I would really appreciate it if you suggested anything.

3

I could use this too. Open CV has a built in tool, but it crashes unity UWP builds