[Tutorial] Using C++ OpenCV within Unity

I tried and i get this error

Error:error: ‘…/…/…/…/src/main/jniLibs/mips64/libopencv_java3.so’, needed by ‘…/…/…/…/build/intermediates/cmake/debug/obj/mips64/libnative-lib.so’, missing and no known rule to make it

after Make Project

it is possible to add as attachment zipped complete project?

Hey @inder2 , have you followed these instructions?

It’s very important that you copy the .so files to exactly that location, otherwise Android Studio won’t find them.

thanks now working i had lib in root not in src

1 Like

thank you for your tutorial @martejpad !

I’m trying yours, and I get “undefined reference on VideoCapture”. It was detected by IDE but seems like it couldn’t be linked.
do you need to include Android.mk & Application.mk as mentioned in OpenCV tutorials? if yes, then how to combine it with build.cradle?


any help appreciated! :slight_smile:

@dani-dev Can you post us the content of your CMakeLists.txt?
I had a similar issue just a few days back and managed to sovle it with marte :smile:

Here is my slightly modified file:

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.4.1)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
native-lib

# Sets the library as a shared library.
SHARED

# Provides a relative path to your source file(s).
src/main/cpp/native-lib.cpp )

find_library( # Defines the name of the path variable that stores the
              # location of the NDK library.
              log-lib

              # Specifies the name of the NDK library that
              # CMake needs to locate.
              log )

# Links your native library against one or more other native libraries.
target_link_libraries( # Specifies the target library.
                       native-lib test-lib
#lib_opencv
                       # Links the log library to the target library.
                       ${log-lib} )

add_library( test-lib SHARED IMPORTED )

set_target_properties(test-lib PROPERTIES IMPORTED_LOCATION

${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libopencv_java3.so)

include_directories(src/main/cpp/include/)

If you copy this and want to use it, make sure to rename your jnilibs folder to jniLibs with a capital “L”.

1 Like

Hey @dani-dev , my guess is that Android Studio is not finding your libopencv_java3.so files, and it’s probably a problem in your CMakeLists.txt file as @Desoxi mentioned. Other option is that you haven’t copied the files to the right location.

Let us know :slight_smile:

1 Like

@martejpad @Desoxi

I’m trying to declare another library (objdet-lib) but it seems didn’t work…

add_library( objdet-lib SHARED src/main/cpp/objdet-lib.cpp )
target_link_libraries( native-lib test-lib
objdet-lib
${log-lib} )

okay, I’m using native-lib instead. and it is running.
now the problem is, my videocapture didn’t work

std::string hello = "Hello from C++";
cv::VideoCapture captest = cv::VideoCapture();
captest.open(0);
captest.open(CV_CAP_ANDROID);
if(captest.isOpened()){
hello += "; video loaded."; //trying to debug it with example
}

do I need to declare OPENCV_CAMERA_MODULES:=ON somewhere? or because I’m using opencv 2.4.11, is it differ from opencv3 on declaring camera?

thanks!

Hey @dani-dev , I’m really sorry but I haven’t tried running Thomas’ example on Android, only on the Editor, so unfortunately I don’t know how to help. I think @Desoxi was also fighting with the VideoCapture part, hopefully he can give you some guidance!

I have working native c opencv code to unity, example is published on Android Play store:
the main problem is release memory from pointer because memory leaks from c plugin profiler dont detect this (currently solved I hope)…

https://play.google.com/store/apps/details?id=com.isenstec.vrbox3

I need some testing on other devices if it is stable, there is image grab from unity send as char array to c opencv → opencv do simple threshold and send back to unity as return char array pointer
after some cleanup i publish whole plugin, later i publish code…

I tested on Samsung S3 working fast and stable… please some tests and compatible devices feedback :slight_smile:


according opencv documentation native grab image by opencv in c code is supported to android 4.X version, some test this way to get image from camera?

if someone need latest library of opencvlib_java3.so for x86 and arm there are i use these:
libopencv_java3 compiled 30.5.2017 from git, opencv download section use old :frowning:

The code will by published there later after some cleanup:
http://193.87.95.129/openvision2/index.php/en/

BRG
Kamil

Hi @inder2 , congrats on the progress! I’ll test your app later today and let you know how it goes :slight_smile:

Regarding the transfer of images between Unity and C OpenCV, what I do to manage the memory is use system pointers like this:

// Allocate memory. ImagePixels is a byte array that contains the pixel values of your image
System.IntPtr pointerIn = Marshal.AllocHGlobal(imagePixels.Length);

// Copy the contents of your byte array to the memory allocated
Marshal.Copy (imagePixels, 0, pointerIn , imagePixels.Length);

// Allocate memory for the image from OpenCV's processing
System.IntPtr pointerOut = Marshal.AllocHGlobal(outImagePixels.Length);

// Call your OpenCV function which looks something like: YourOpenCVFunction (unsigned char * inImage, unsigned char * outImage) on the C side.
YourOpenCVFunction (pointerIn, pointerOut);

// Copy the result of the processing in a byte array
Marshal.Copy (pointerOut, outImagePixels, 0, outImagePixels.Length);

// Free system pointers
Marshal.FreeHGlobal(pointerIn);
Marshal.FreeHGlobal (pointerOut);

This is the way I’ve found the memory handling works for me, but I’m sure there must be more efficient ways to do it. How do you currently do it?

almost same but i get image back without out function by return:)
ptr2 = ocv_get_image(resx, resy, pixelsHandle.AddrOfPinnedObject());

i had memory leaks i used separate function release_memory from c;
i will try your way Marshal.FreeHGlobal(pointerIn); i think is better :stuck_out_tongue:

1 Like

Stumbled across your guy’s work just now, you guys are amazing! Tested it just now on my Android Nexus 6P and it’s functional. Since you were asking about feedback, I couldn’t get the “overlay”,“change”, or “test vr2” buttons to do anything. Also, I assume this is just not implemented yet, but the feed does not (counter) rotate to accommodate landscape or profile rotations.

@martejpad : what are these two lines in the cmake file doing?
add_library( test-lib SHARED IMPORTED )
set_target_properties(test-lib PROPERTIES IMPORTED_LOCATION

and since you named your so file as:
“libnative-lib.so”

are the methods accessed using:
[DllImport(“libnative-lib”)]
internal static extern FUNCTION_NAME();

Thanks

“test vr2” buttons to do anything. Also, I assume this is just not implemented yet, but the feed does not (counter) rotate to accommodate landscape or profile rotations.

test vr2 currently not working i want implement camera video to vr google cardboard but there some problem (i solving this now)…
application crash… if i enable vr :frowning:

Hi @alemfi538 ! Those two lines are basically telling the compiler to add OpenCV’s SO library. “test-lib” was a really poor choice for the name of the lib, it should be something like:

add_library( opencv-lib SHARED IMPORTED )

set_target_properties(opencv-lib PROPERTIES IMPORTED_LOCATION
     ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libopencv_java3.so)

With regards to the DllImport, you can use [DllImport(“native-lib”)], without the “lib” prefix. Have a look at Thomas’ tutorials for a few good examples of usage.

Hope this helps!

Thanks for the response @martejpad , I did read up that I need to remove the lib prefix, however I’m still encountering some difficulties accessing the library (or perhaps the functions?)

Uploading an image with the hierarchy and inspector of the .so files on the left, in the middle are some attempts to declare exports of the C++ code, and on the right hand side is attempts to access those functions from the C# code. Not sure if I’m overlooking something as it is currently 3am my time and I should really be getting some shut-eye

3095454--233624--2017-06-05.png

Hey @alemfi538 ! Could you please elaborate a bit more on the difficulties you’re encountering? Do you get any error messages? Are you getting any image back from OpenCV?

Looking at your code I’m not sure if the way you handle the memory (returning a char * from OpenCV) is the safest, @inder2 was having some memory leaks doing it like this. Maybe it’s worth trying pre-allocating the memory in C# beforehand and see if that’s the problem? Can you show us how you’re calling the function?

If you give me more details regarding your issues I’ll try to think of possible solutions. But go get some sleep! I’m sure thinks will be clearer later :slight_smile:

Regards

Oh yes, no problem. In android studio, after performing the attempt to call the Hello World Function I am getting an error/exception:

“06-05 02:57:04.149: E/Unity(10932): Unable to find native-lib”

Running it outside of the try allows me to get a more detailed stacktrace:
06-05 13:24:30.214: I/Unity(20290): DllNotFoundException: native-lib
06-05 13:24:30.214: I/Unity(20290): at (wrapper managed-to-native) DisplayFeed:HelloWorld ()
06-05 13:24:30.214: I/Unity(20290): at DisplayFeed.Start () [0x0006f] in D:\Documents\Interviews\SnapChatTest\SnapChatInteractivityTest\Assets\Scripts\DisplayFeed.cs:47

I am wondering, since the “.so” files were built as “Debug” and not “Release”, did you do a development build in Unity? or should that not matter?

Oh no. I think I just realized… did you need to include the : “libopencv_java3.so” built previously into the unity project as well? I believe I only added the newly built libnative-lib.so libraries. I think I might have fixed the issue. Possibly.

Edit: Got it to work! it was indeed because I did not include the previously built opencv libraries. One thing to note is that the camera appears to be rotated 90 degrees (for me) Thanks a bunch!

Has anyone figured out where to store the cascade files when building to android? just hit a small snag where it can’t locate the files.

So glad it’s working @alemfi538 !
With regards to the cascade file, in theory if you put it under Unity’s Streaming Assets it should get copied to the target device, have you tried that? I know that @Desoxi had a bit of an issue getting it to work, I think he ended up just copying it manually to the device and hard coding the path, but I can imagine that’s not an ideal solution :slight_smile: I haven’t tried myself, sorry for not being more helpful!

@martejpad
So it turns out that on Android the streaming assets get stored in a compressed format (https://docs.unity3d.com/Manual/StreamingAssets.html) so I had to extract them from Unity, write into a new file at Application.persistentDataPath so that the new file’s path can be accessed by opencv (opencv cannot read from the compressed file).

Still need to figure out the 90 degree issue, but I think that’s most of the technical leg work done. For now.

@alemfi538 Good to know that detail about the Streaming Assets folder, thanks a lot for the info. I’m glad you found a workaround! Please keep letting us know how everything goes, we’re all learning here :slight_smile: