[Tutorial] Using C++ OpenCV within Unity

Unity Engine
1

Hi there,

I required some OpenCV image processing for a project a couple of months ago, and didn’t really find a lot of information on that subject. I also did not want to buy an asset for a free library. In an effort to help other developers in the same situation, I decided to write a couple of tutorials to kick-start their implementation of OpenCV within Unity:

Note that these are not OpenCV tutorials, but focus only on getting OpenCV to communicate with Unity.

Hopefully this’ll be of use to some of you! Let me know if you have any questions on the matter at hand.

20 Likes
2

Hi thomas, i was asked to investigate existing solutions about openCV and Unity3D.
I found your tutorials and i followed them, encountered some trouble at some point but that were my mistakes and your instructions were clear.
I finally got to the end, created the two scripts, put them on a gameObject andi tried on the camera but i can’t see anything happening on my screen. Maybe you can tell me, what should i do when i created the two scripts? i pluged in a webcam and unplugged it, i can’t see any camera flux. I guess i am not understanding something.
I just followed everything quite blindly to be frank as i don’t know anything on openCV itself. I am a unity3D developper.
I created the two scripts. I had an error on the dll import because my dll was not named “UnityOpenCVSample” so i replaced by mine and it worked.

I only have this log now when i play the scene. I took the “lbpcascade_frontalface.xml” from the source files and copied it at the root, i tried putting it in the StreamingAsset folder but it doesn’t change
“[OpenCVFaceDetection] Failed to find cascades definition.
UnityEngine.Debug:LogWarningFormat(String, Object[ ])
OpenCVFaceDetection:Start() (at Assets/OpenCVFaceDetection.cs:28)”

Hope you can answer me !
Great work !

3

Hey Pirou,

In Part 3, the face tracking is only used as an input to drive a normalized screen position, which can then be used to move objects around. There is only a debug camera output displayed in an OpenCV imshow() window, the camera output is not displayed within Unity. The OpenCVFaceDetection script goes onto an empty game object, and the PositionAtFaceScreenSpace sample script goes onto a mesh in front of the camera (down the z axis).

However, in your case, nothing is happening because OpenCV did not find the cascade defintion. Did you place it in the root project folder (NOT the root Assets folder)?

In the upcoming part 4, I will show how you can use the same fixed pointer technique to pass an array of pixels to Unity so you can have a texture with the OpenCV camera output.

4

Hi thomas, thanks for the quick reply,
on my own i solved the file not found and it’s true that i put it in the assets folder and not the root project (my bad, i thought the root folder meant the asset folder)
i search online and from what i found, i guessed the path was wrong so i replaced the name of the file by my own path in the code in visual studio, compiled it again and replaced the dll. The xml was found but i launched the unity scene,a small window is launched named “Unity OpenCv Interop Sample” and i see a black screen. i guess i didn’t get everything or i am not supposed to see anything !

2994178--223015--launch.JPG
2994178--223015--launch.JPG1907×1020 147 KB

5

That window is the OpenCV imshow() window (it’s the last line of the OpenCV code). It should contain your camera output and the face detection. It seems however that the camera stream is not working on your end - can you try just having a bare bones OpenCV camera application to see if it is able to read your camera? You can use this sample code in a new C++ project, set up as shown in part 2.

6

i am gonna try. Keep you posted

Edit: can’t seem to make it work. I created a new project, set up exactly like the other following the part 2 but when i copy the sample code, i got some errors with the library so i modified and now it doesn’t recognize VideoCapture type

Edit2: i don’t know why but i had some huge trouble launching on VS studio the code you told me to test but i finally made it work. I guess i corrupted something during my tests on VS. I generated everything again with cmake and it works on VS ! I am gonna try to call it from unity3D

Edit3: i can’t make the call from unity3D,i am told the dll is not found but it’s in the plugins folder.
Maybe i need to change the code and not try to call the main?

7

I did it ! I was able to call functions from the dll inside unity and run the simple example displaying the camera.
I made some tests and i can only call functions returning void or int but when i tried with string, the unity crashed !
My friends worked with openCv before so i am gonna try to run more complex examples !

8

Great tutorial. Thank you Thomas. I wasn’t getting frames displayed either.
To fix that, I changed _capture.open(-1) to _capture.open(0) in the Init native function. That did the trick. Works like a charm.

2 Likes
9

Great tutorial @Thomas-Mountainborn and thanks for the tip @DanielBernier !

I am close to getting this working correctly. I can get the capture feed to display perfectly without doing any facial detection. When I start attempting to do the detection, however, Unity crashes. I’ve narrowed it down to this line:

_faceCascade.detectMultiScale(resizedGray, faces);

If I comment this out, everything works fine … not sure what’s going on exactly. The error log says:

opencv_objdetect320d.dll caused an Access Violation (0xc0000005)
in module opencv_objdetect320d.dll at 0033:5d494ce7.

I’m going to keep looking into it but if you have any suggestions, please advise. Thanks again for taking the time to make the tutorial(s)!

10

Hey jsfledd, I don’t know why you’re getting that error there. I do notice that you’re using the debug .dll, whereas the tutorial is using the release versions - make sure that you’re either using all debug .dll’s and build your package as Debug, or all release .dll’s (no “d” suffix in the OpenCV filenames) and build your project as Release.

Also Daniel, I updated the tutorial to your feedback.

1 Like
11

That was it! Thanks Thomas. I had both the debug and release dlls in there at the same time.

1 Like
12

(this is my first post on the forum, please bear with me :slight_smile: )

Amazing tutorial @Thomas-Mountainborn , thanks so much for taking the time to put it together!

I wanted to ask you whether it’s possible to send an image (or texture) to OpenCV, and receive the image back in Unity after doing some processing. I’m using Vuforia within Unity so I need to make sure the image processed by OpenCV is exactly what’s captured and displayed by Vuforia’s AR camera, that’s why I don’t want to do the image capture inside OpenCV.

You mentioned in a previous answer that you will be showing in part 4 how to send a pixel array from OpenCV to Unity, so perhaps that’ll answer my question :slight_smile:

Thanks so much for your help again, it’s super appreciated!!

1 Like
13

Hey martejpad, that’s indeed the thing that will be discussed in the next part. The exact same technique will be used: a pointer to an array of pixels is passed in between Unity and OpenCV.

1 Like
14

Many thanks for the super quick response! Really looking forward to part 4 and learning how to do it - when can we expect it? (no pressure :roll_eyes:)

15

Well, I start every weekend with the intention of writing it, but you know how it goes. I’ll see if I can dig up some snippets to set you on your way sooner.

3 Likes
16

Thanks a million Thomas, that would be super helpful!!

After a lot of sweating I’ve finally managed to get your sample to work. I’ve followed your instructions but adapted to MacOS, Monodevelop and Eclipse (I’m brave I know…!). Everything works fine although every time I stop playing the scene Unity crashes, but for now I’m going to ignore that…

The next natural step would be to make this work on Android or iOS. Could you possibly shed some light on how to do this? I’ve tried Building and Running this example for Android but I get a NullReferenceException error (I imagine VideoCapture just doesn’t work).

Apologies if this is super trivial or I’m talking nonsense, I’m really new to this!

Thank you so much again for willing to help all of us and being so generous with your time!!

17

Hi! Just wanted to share that I’ve managed to use OpenCV (natively in C++, not Java) in Unity from an Android app :smile: I used Android Studio to compile an .so library, if anyone is interested in the details of the process let me know and I’m happy to share everything!

Thanks!!

18

Please share tutorial unity andriod opencv

19

how to use OpenCV in Unity to create an Android App?

20

Hey guys,

I’m going to share here my personal write-up of the process I followed to develop natively (C++) using OpenCV, then use this code to build an Android app from Unity. I use MacOS. Big disclaimer: I don’t have a background in software development so I can’t guarantee the correctness of any of this. All I know it’s that it’s working for me right now :slight_smile: I also recommend you have a read of all the resources I link, they helped me a lot.

The process Thomas shared is valid to compile a native (C++) OpenCV code into a shared library to be read from the Unity Editor and in a Windows machine, although it can be easily adapted for MacOS, but that’s a different story. If you want something that works on Android, you instead need to cross compile your code in the form of an .so library.

Note: In this SO post (Answer by Bilelmnasser) there’s a brief explanation of how the libraries need to be loaded from Unity according to the target platform. And here’s the information Unity provides regarding developing native plugins for Android.

The main thing to understand (sorry if this is too basic for you) is that when one downloads the regular OpenCV version, this is meant to be compiled for your machine’s OS (Windows, Linux, MacOS). For example in my case (MacOS) this means that the compilation results in a set of .dylib libraries that are installed in /usr/local/include and /usr/local/lib, and can be used as explained by Thomas in his tutorials. If instead we want to be able to run OpenCV from an Android platform, we need to use OpenCV4Android, which allows development of OpenCV code for Android, both natively in C++ and in Java (by using the provided wrapper). OpenCV4Android can be downloaded from their website (I specifically downloaded this package - 3.2.0 version).

If like me you’re interested in developing C++ OpenCV code, you can follow the process I’m about to share. If instead you want to use Java OpenCV, this SO post (don’t read the verified answer, but instead the one that follows) contains a brilliant step by step for setting up Android Studio for developing Java OpenCV. I tried it and it works.

Android allows the use of native code in its apps by means of their NDK toolset. This toolset is available to download on its own, here (and then it can be used in any IDE or through command line), or it can be downloaded through the SDK manager inside Android Studio, as explained here. I chose to use the NDK within Android Studio as it seems to be the recommended way. This NDK essentially allows you to cross compile native code that can be run on an Android platform, which is exactly what we need.

OpenCV offers very messy tutorials of how to use OpenCV4Android. Available resources are gathered on this page. The main problem is that the tutorials are outdated. In the case of developing in Java it’s all based on the deprecated ADT plugin for Eclipse, which gave GUI-based access to many of the command-line Android SDK tools. On the other hand, in the case of native OpenCV development, these tutorials offer instructions on how to use the NDK within Eclipse and use ndk-build, which is slowly being replaced by CMake (I think NDK support is now directly built in CMake, see here). I think these instructions are still valid, but I preferred to opt for the “recommended” way of using the NDK directly from Android Studio and using CMake.

The key things that are inside OpenCV4Android folder (info from this tutorial from OpenCV’s website):

⇒ sdk/java folder is key if you’re going to use Java OpenCV code, since it contains the Java wrapper. It can be found in sdk/java/src/org/opencv. We’re not going to use the contents of this folder since we’re going to use C++ instead.

⇒ sdk/native folder contains OpenCV C++ headers (for JNI code) and native Android libraries (*.so and *.a) for ARM-v5, ARM-v7a and x86 architectures. Header files are in sdk/native/jni/include and libraries in sdk/native/libs.

Assuming you have now have OpenCV4Android and Android Studio + NDK, now we can move on to how I’ve done the whole process in Android Studio. Reading these instructions really helped me to figure this out.

⇒ Create project with C/C++ support

⇒ Check “Phone and tablet”, leave the recommended minimum SDK version.

⇒ Include an Empty Activity (this will create files necessary for the C++ functionality)

⇒ Keep the rest of the options as they come by default.

⇒ Android Studio has now created a project folder. Inside it, in app/src/main/cpp/, Android Studio will have created a file called native-lib.cpp, which contains an example function called stringfromJNI() that we can ignore. However, you can write your C++ OpenCV code in this file.

⇒ In the app folder inside your project folder, a file called CMakeLists.txt will be created. This is the file with all the instructions on how to compile your native C++ code. Leave it as it is right now, we’ll be modifying it in a bit.

⇒ Copy the libraries of OpenCV4Android folder, contained in sdk/native/libs (all folders for different architectures) in app/main inside your Android Studio project. Change the name of the parent folder from libs to jniLibs. Delete all static libraries (.a) from all folders, only leaving libopencv_java3.so in each of the architecture’s folders. The other ones are not needed.

⇒ Copy the header files in OpenCV4Android (sdk/native/jni/include) to a folder in app/src/main/cpp. I opted to name the parent folder include as well. I think this folder could be anywhere, the only important thing is that it needs to be in a location accessible by Android Studio.

⇒ Edit the CMakeLists.txt file so that 1) it imports the OpenCV library (libopencv_java3.so) as a shared library; 2) it adds the library as a target; 3) it includes the path to OpenCV’s header files. This is a copy of the file I’m using, which does all of this.

For more information about using CMake with Android Studio, read the

documentation: Add C and C++ code to your project  |  Android Studio  |  Android Developers

Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.4.1)

Creates and names a library, sets it as either STATIC

or SHARED, and provides the relative paths to its source code.

You can define multiple libraries, and CMake builds them for you.

Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
native-lib

Sets the library as a shared library.

SHARED

Provides a relative path to your source file(s).

src/main/cpp/native-lib.cpp )

add_library( test-lib SHARED IMPORTED )

set_target_properties(test-lib PROPERTIES IMPORTED_LOCATION

{CMAKE_SOURCE_DIR}/src/main/jniLibs/{ANDROID_ABI}/libopencv_java3.so)

include_directories(src/main/cpp/include)

Searches for a specified prebuilt library and stores the path as a

variable. Because CMake includes system libraries in the search path by

default, you only need to specify the name of the public NDK library

you want to add. CMake verifies that the library exists before

completing its build.

find_library( # Sets the name of the path variable.
log-lib

Specifies the name of the NDK library that

you want CMake to locate.

log )

Specifies libraries CMake should link to your target library. You

can link multiple libraries, such as libraries you define in this

build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.

native-lib test-lib

Links the target library to the log library

included in the NDK.

${log-lib} )

Note: You can ignore the log-lib stuff, it’s just there because it was in the sample of Android studio.

⇒ Now the project is ready to be built in Android Studio: Build->Make project. This will generate a shared library (.so file, in my case called libnative-lib.so) for our native code for each architecture in jniLibs. The generated libraries can be found in the folder app/build/intermediates/cmake/debug/obj.

⇒ Now in Unity! Create a folder called Plugins inside the Assets folder. Then another called Android inside Plugins, and another called libs inside Android. Copy the folders “x86” and “armeabi-v7a” from app/build/intermediates/cmake/debug/obj. These are the processor architectures that Android supports (ARMv7 and x86), see here for more info. Android also supports MIPS but it’s the least popular and not supported by Unity. Also, the 64 counterparts of ARM and x86 are not supported by Unity either. When we later build the app in Unity, it generates a FAT APK by default, which works in both architectures. This setting can be changed in Build Settings->Player Settings ->Android->Other settings->Device filter. IMPORTANT EDIT: Also copy inside the corresponding architecture folder in Plugins the file libopencv_java3.so that can be found in OpenCV4Android/OpenCV-android-sdk/sdk/native/libs.

Once you’ve copied your .so in this folder, Unity will treat them as plugins → see here for details on the Plugin inspector and the different file extensions that are treated as plugins by Unity. The folder structure is very important as detailed in the Deployment section of this article, so Unity picks up the right library according to the target architecture (mmm).

Important resources provided by Unity:

=> Finally, to use the C++ functionality from Unity, you can follow Thomas’ tutorials, in particular Part 3.

Note: If you don’t need to develop natively, I believe there’s also the possibility to use OpenCV directly from Unity with Emgu CV, which is a cross platform .Net wrapper to OpenCV, allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc. To make this work follow this SO post (haven’t tried it).

Let me know if this works for you!

9 Likes