Opencv 4.1.1 as DLLImport Reimport from Image does not work with memcopy

Unity Engine
,
1

Hi,

i’m currently working on saliencymap creation in Unity an image. I got the opencv libary up and running, but i’m not able to send the saliencymap from c++ to c#.
I added two screenshots for the visual aspect of my problem.

i have the following code in use:

extern "C" bool __declspec(dllexport) __stdcall ComputeSaliency(unsigned char* image, int width, int height, unsigned char* saliencyMap)
{
    Mat matSaliencyMap(height, width, CV_8UC4);
    Mat matImage(height, width, CV_8UC4, image);
   
    flip(matImage, matImage, 0);
    cvtColor(matImage, matImage, COLOR_BGRA2RGB);
   
    Ptr<Saliency> salAlg;
    salAlg = StaticSaliencyFineGrained::create();
    salAlg->computeSaliency(matImage, matSaliencyMap);

    Mat saveSaliency;
    matSaliencyMap.convertTo(saveSaliency, CV_8UC4, 255);
    imwrite("D:\\Unity Projekte\\LabelPlacement\\Assets\\Resources\\saliencyMap.png", saveSaliency);

    flip(saveSaliency, saveSaliency, 0);               
    memcpy(saliencyMap, saveSaliency.data, saveSaliency.total());
}

and for c# in Unity

public Texture2D SaliencyMapWithOpenCV(Texture2D newTexture) {
        Texture2D tmpTexture = new Texture2D(newTexture.width, newTexture.height);
        Color32[] pixel = newTexture.GetPixels32();
        Color32[] saliencyMap = new Color32[pixel.Length];

        handlePixel = GCHandle.Alloc(pixel, GCHandleType.Pinned);
        handleSaliency = GCHandle.Alloc(saliencyMap, GCHandleType.Pinned);

        ComputeSaliency(handlePixel.AddrOfPinnedObject(), newTexture.width, newTexture.height, handleSaliency.AddrOfPinnedObject());
       
        for (int i = 0; i < saliencyMap.Length; i++) {
            saliencyMap[i].a = 255;
        }

        tmpTexture.SetPixels32(saliencyMap);
        tmpTexture.Apply(true);

        return tmpTexture;
    }

and finally the sprite creation with the texture in c# as well

saliencyTexture = si.SaliencyMapWithOpenCV(texture2D);
saliency.sprite = Sprite.Create(saliencyTexture, new Rect(0.0f, 0.0f, saliencyTexture.width, saliencyTexture.height), new Vector2(0.5f, 0.5f), 100.0f);

Thanks you in advance,
Basti

4841111--465095--c# texture Image.png

4841111--465098--Saliency Map in c++.png
4841111--465098--Saliency Map in c++.png735×554 147 KB

2

I see you pin those handles but I’m pretty sure you need to let them go before you can get the data back from them on the managed side. That’s the point of it all.

The construct I use (and I might have posted this to you in a previous post?) is:

        pixels = new Color32[ pixelSpan * pixelSpan];
        GCHandle handle = GCHandle.Alloc(pixels, GCHandleType.Pinned);
        try
        {
            dispatcher1_rendercolor( handle.AddrOfPinnedObject(), span, 0);
        }
        finally
        {
            if (handle.IsAllocated)
            {
                handle.Free ();
            }
        }
        t2d.SetPixels32( pixels);
        t2d.Apply();
3

Hi,
yeah you answered me this in a previous post. I had the Free() function in OnApplicationQuit, for some reason. But if I use your conrstruct i have the same problem. The copy from c++ to c# still creates a way smaller image and 4 times result image. I’m stuck right now, as for others there is no problem with memcpy the processed image to a unity Color32[ ] array.

try {
ComputeSaliency(handlePixel.AddrOfPinnedObject(), newTexture.width, newTexture.height, handleSaliency.AddrOfPinnedObject());
} finally {
if (handleSaliency.IsAllocated) {
handleSaliency.Free();
}
if (handlePixel.IsAllocated) {
handlePixel.Free();
}

}
4

I’m not sure what this means. Is it just de-interleaving the individual color channels and presenting it? Does the saliency computation intentionally return a lower-resolution image?

5

Hi,

i attached two pictures of my problem in the first post in here. I have the map correct in opencv. I even can save the file in the correct resolution. But when i memcpy the image to my Color32[ ] array and create a texture with it and with the texture a sprite. The sprite is the map 4x times in the bottom row and 2/3 of the image is black.

6

Hey,
for later use: i just needed a propper conversion to get the image back to unity!

cvtColor(matSaliencyMap, matSaliencyMap, COLOR_RGB2BGRA);
matSaliencyMap.convertTo(matSaliencyMap, CV_8UC4, 255);
flip(matSaliencyMap, matSaliencyMap, 0);
memcpy(saliencyMap, matSaliencyMap.data, matSaliencyMap.total() * matSaliencyMap.elemSize());

Thanks for your help again! :slight_smile:

1 Like