OpenCVSharp for Unity

NWHCoding · October 8, 2017, 11:49am

Fast and easy to use asset that brings OpenCV functionality to Unity.

Features:

Supports following platforms: Windows, Linux, Mac OS X, Android and iOS.
Works both in editor and as a standalone.
Includes fast functions for easy conversion between common Unity and OpenCV structures (WebCamTexture to Mat, Mat to Color32[ ], etc.). No conversion is done iteratively.
Uses vanilla C++ OpenCV 3.3.1 with C# wrapper and syntax.
Due to the direct calls from C# to the C++ native OpenCV plugins 60+ FPS is easily achievable on mobile, and 200+ FPS on desktop devices.
IDisposable is implemented along with exception redirection to standard Unity errors.

Conversion functions:

MatToTexture2D
MatToColor32Array
Texture2DToMat
Texture2DToColor32
Texture2DToRenderTexture
Color32ArrayToMat
Color32ArrayToTexture2D
Color32ArrayToByteArray
ByteArrayToColor32Array
RenderTextureToTexture2D
WebCamTextureToMat
WebCamTextureToTexture2D
WebCamTextureToRenderTexture

Conversions are mostly made through memory copy or similar methods and work on all supported devices - no iterative conversions. RenderTextureToTexture2D conversion also allows use of OpenCV on in-game cameras - such as the line following robot example.

Included modules:

core
calib3d
imgproc
shape
superres
stitching
video
aurco
bgsegm
dnn
face
img_hash
optflow
tracking
xfeatures2d
imgproc
xphoto
text

Included helper functions:

GetWebCamTexture2D - returns image from webcam as Texture2D.
GetWebCamMat - return image from webcam as OpenCV mat.
GetIntPtr - returns pointer to the structure.
GetStreamingAssetsPath - returns path to the file in the Streaming Assets folder. Platform independent. Useful for ImRead and similar functions that require path to the file.

More info:
Website
Youtube Channel

Demos:
Windows (64-bit)
Windows (32-bit)
MacOS
Android
iOS

For more info and screenshots check out OpenCVSharp for Unity website.

Asset is now available on the Unity Asset Store. Click here to visit.

ZJP · October 25, 2017, 9:52pm

On the Store.
Bookmarked.

r618 · December 5, 2017, 2:02pm

According to README:

this is not true - except StreamingAssets folder - since Unity 5.something - Plugins folder can be placed anywhere below root Plugins folder and Unity will consolidate all of them.
I like to keep things organized so I just moved the whole OpenCVSharpForUnity folder into Plugins - works okay.
But the main caveat after importing is to set manually each respective native dll to correct architecture:

otherwise the Editor will complain it can’t distinguish between same named libraries and example scenes won’t work. ( this is 2017.2 )

This should be mentioned in the documentation, or enforced on import ( don’t know if asset store package upload preserves meta file import setting tbh )

On Android with some landscape orientation I had camera image upside down and Hough lines sample reliably crashed the app. Haven’t tested on iOS yet.

But other than the above I’m really glad you chose to all native opencv libraries, and include sources for iOS - performance is great !

Cheers

NWHCoding · December 5, 2017, 2:07pm

r618:

According to README:

this is not true - except StreamingAssets folder - since Unity 5.something - Plugins folder can be placed anywhere below root Plugins folder and Unity will consolidate all of them.
I like to keep things organized so I just moved the whole OpenCVSharpForUnity folder into Plugins - works okay.
But the main caveat after importing is to set manually each respective native dll to correct architecture:

otherwise the Editor will complain it can’t distinguish between same named libraries and example scenes won’t work. ( this is 2017.2 )

This should be mentioned in the documentation, or enforced on import ( don’t know if asset store package upload preserves meta file import setting tbh )

On Android with some landscape orientation I had camera image upside down and Hough lines sample reliably crashed the app. Haven’t tested on iOS yet.

But other than the above I’m really glad you chose to all native opencv libraries, and include sources for iOS - performance is great !

Cheers

Hello,
sorry for the confusion. I believe that the switch happened when going from 5.x to 2017.x. I initially made it for 2017.1 and then ported it back to 5.6 to ensure backwards compatibility. I will need to fix this on both this asset and Grapher. I will indeed have to mention this in the manual so people with newer versions of Unity do not get confused.
I am also glad you like the rest, development and building for each platform was hell and took me a lot of time.

r618 · December 5, 2017, 7:02pm

@NWHCoding unfortunately link step on iOS fails for me:

(with practically everything getting unresolved afterwards)

strangely, my attempt to specify just armv7 in target build settings fails too:

I don’t know what exactly this means for the format of frameworks to be compiled as ( I’ve mistakenly thought that iOS frameworks are distributed as sources - all the hpp’s mislead me o), but it seems that something is missing in them currently for Xcode to be satisfied
this is latest Xcode 9.2 compiling for 11.2

NWHCoding · December 5, 2017, 7:19pm

Uh, oh. I have built for iOS just two days ago on High Sierra with XCode 9.2 and the build went without a problem. Granted, I do not have an actual iOS device to test it on, but I sure did not get the message. iOS frameworks (unfortunately) are not distributed as source (which would be nice). They need a lot of building actually, the headers are there for Unity. I have a feeling this might be something related to build settings and/or iOS version.

Here is a screenshot from two days ago:

And here are the settings:

Also, in the folder Plugins/iOS there are 3 files and each should have iOS ticked. I doubt this is the problem since Unity seems to find correct plugins.

Could you please try and build with those settings? I am on a virtual machine and my build will take some time. If you want you could send me your settings and I will try to build with those. iOS is very picky about everything.

EDIT: Also tried setting 11.2 as target and it passed.

r618 · December 5, 2017, 7:45pm

I’ve been building iOS stuff since unity 1.6, of course I have all basics set and am providing support for my own iOS integration, occasionally ]

Have you ever built for and run on an actual iOS device ?

build for Generic iOS device in Xcode fails with the same linker errors

Everything it would take to reproduce this is getting current version of the package from the store, import into new project, build for iOS, and build Xcode project.
Let me know if you are able to repro this.

r618 · December 5, 2017, 7:57pm

Ah, according to file OpenCvSharpExtern is indeed missing arm architectures:

opencv2 itself:

so that would probably explain it - now to actually build the extern fat library is all that is left
Good luck and don’t stress about this - just let me know once update is ready.
Cheers!

NWHCoding · December 5, 2017, 8:12pm

Thank you for the spot, since I do not own the actual device I managed to oversee this - simulator only supports i386 and x86_64 which is a bit lame. Will fix it tomorrow as it is getting a bit late now and send you the fixed package. Asset Store update will follow as soon as admins approve it.
Once again thank you for the bug report and sorry for the problems.

NWHCoding · December 6, 2017, 4:20pm

Missing architecture problem for iOS (device) has been fixed and will be released as an update 1.1.
It will be pushed to the store tomorrow which means it will be available for download from the store in a few days, after it has been approved.
If you need it sooner send me a PM or email and I will send it to you.

NWHCoding · January 4, 2018, 9:30pm

Price has been reduced to $28 for the time being.

NWHCoding · March 2, 2018, 8:44pm

New update coming during the next week. Asset will be updated to OpenCV 3.3.1 and newest OpenCvSharp wrapper.

NWHCoding · March 7, 2018, 6:50pm

Update has been finalized and beta is available. Asset Store release will be beginning of the next week.
If anybody wants to try it out sooner send me a PM or an email with your invoice.

ZJP · March 8, 2018, 2:51pm

Hi,

Do you have any examples with ArUCo marker detection?

NWHCoding · March 8, 2018, 3:16pm

No, only the examples from the Menu scene are included. Still, this asset uses OpenCvSharp wrapper so any code written for it will work here too, and porting from C++ is not hard since syntax is quite similar (not same tho).

ZJP · March 13, 2018, 3:09pm

Thanks for the reply.

mrmoonshield · April 14, 2018, 6:18pm

Hi, in opencv TermCriteria for optical flow is written as follows:
TermCriteria termcrit(TermCriteria::COUNT | TermCriteria::EPS, 20, 0.03);
how does it declare in OpenCVSharp?

system · May 3, 2018, 11:43am

Gaussian and Median blur is crashing.

Omini · May 5, 2018, 3:01am

Unity crashes when I try to draw markers. Any idea why? Here is my code:

using System.Collections;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using UnityEngine;
using OpenCvSharp;
using NWH;
using UnityEngine.UI;
using OpenCvSharp.Aruco;
public class Marker : MonoBehaviour {
    private RawImage rawImage;
    private WebCamTexture webCamTexture;
    private Texture2D tex;
    private Mat mat, gray;
    Dictionary dictionary;   
    void Start()
    {
        dictionary = CvAruco.GetPredefinedDictionary(PredefinedDictionaryName.Dict5X5_50);
        rawImage = GameObject.Find("Canvas/RawImage").GetComponent<RawImage>();
       
        webCamTexture = new WebCamTexture(WebCamTexture.devices[0].name);
        webCamTexture.Play();
        mat = new Mat(webCamTexture.height, webCamTexture.width, MatType.CV_8UC3);
        gray = new Mat(webCamTexture.height, webCamTexture.width, MatType.CV_8UC1);
        tex = new Texture2D(webCamTexture.width, webCamTexture.height, TextureFormat.RGBA32, false);
    }
    void Update()
    {               
        if (webCamTexture.didUpdateThisFrame && webCamTexture.isPlaying)
        {
        CamUpdate();
        }
    }
    void CamUpdate()
    {
        CvUtil.GetWebCamMat(webCamTexture, ref mat);
        int[] ids;
        Point2f[][] corners;
        Point2f[][] rejects;
        InputArray markers;
        DetectorParameters parameters = DetectorParameters.Create();
        markers = InputArray.Create(gray);       
        CvAruco.DetectMarkers(markers, dictionary, out corners, out ids, parameters, out rejects);
        CvAruco.DrawDetectedMarkers(mat, corners, ids);       
        CvConvert.MatToTexture2D(mat, ref tex);
        rawImage.texture = tex;
    }
    private void OnDestroy()
    {
        webCamTexture.Stop();
    }
}

NWHCoding · May 5, 2018, 8:38am

Could you paste your code?

Topic		Replies	Views
[RELEASED] OpenCV for Unity Community Showcases Asset-Store-Assets	3291	517014	May 15, 2025
[RELEASED] OpenCV plus Unity: OpenCvSharp port to Unity Community Showcases Asset-Store-Assets	7	5512	February 23, 2021
Convert Texture2d to opencvsharp to Mat (and vice versa) Unity Engine Post-Processing	2	7124	February 15, 2021
OpenCVSharp for Unity Community Showcases	21	29385	February 15, 2018
Using OpenCVSharp in Unity 5 Unity Engine Scripting	7	1459	August 25, 2016

OpenCVSharp for Unity

Related topics