[RELEASED] OpenCV for Unity

Community Showcases
2561

I will try!!! thanks a lot!

2562

it’s work fine on a picture but i would like try this with webcamtexture…

2563

CV vtuber example, it’s working fine except the video capture example; I can’t get it to work. In fact, any of the video capture examples (OpenCV or DlibFaceLandmarkDetector video capture examples).

ArgumentException: imgMat.isContinuous() must be true.
CVVTuber.OpenCVForUnityUtils.SetImage (DlibFaceLandmarkDetector.FaceLandmarkDetector faceLandmarkDetector, OpenCVForUnity.CoreModule.Mat imgMat) (at Assets/CVVTuberExample/CVVTuber/Scripts/Utils/OpenCVForUnityUtils.cs:32)
CVVTuber.DlibFaceLandmarkGetter.UpdateValue () (at Assets/CVVTuberExample/CVVTuber/Scripts/DlibFaceLandmarkGetter.cs:216)
CVVTuber.CVVTuberControllManager.Update () (at Assets/CVVTuberExample/CVVTuber/Scripts/Core/CVVTuberControllManager.cs:52)

2564

Could you tell me the environment you tested?
OpenCVForUnity version :
DlibFaceLandmarkDetector version :
CVVTuberExample version :
Unity Version :
Editor Platform :

2565

OpenCVForUnity: 2.4.2
DlibFaceLandmarkDetector version :1.3.2
CVVTuberExample version :1.0.6
Unity Version : 2019.4.18f1
Editor Platform : windows 7 sp1.

2566

The system requirement for OpenCVForUnity is Windows 8 or later. Does the error occur in Windows 8 or later?

2567

I seem to have “found” a bug in the plugin:
Defining and initializing a MatOfDouble inside a class works if the Unity Editor is already running but if I exit Unity it crashes on startup.

The line causing the crash is (dummy code):

public class abc : MonoBehaviour
{
     private MatOfDouble mat = new MatOfDouble(new double[] { 0, 0, 0, 0 });
}

This code works without issues if I add it in the script while Unity is running but if I exit the Editor and then start it again, with no changes in the previously working script, it crashes on startup.

If I initialize the variable in e.g. the Awake function I can close Unity and open it again without issues (dummy code):

public class abc : MonoBehaviour
{
     private MatOfDouble mat;

     void Awake() {
          mat = new MatOfDouble(new double[] { 0, 0, 0, 0 });
     }
}

Looking into the Editor.log created by Unity during the crash this is the core error message:

After a short Google search a similar issue seemed to exist in a different project: 0001538: Unity Crash on Project Load - NoesisGUI Issue Tracker
As their workaround is basically the same that I came up with and they diagnosed an issue in their project I think this also is a bug in the plugin and not in my code or Unity itself.

I was able to reproduce the issue in Unity 2019.4.17f1 and 2019.4.18f1 with Windows 10 and plugin version 2.4.2.

2568

Hi,
I am working with the Yolo Object Detection example. I am trying to figure out how to overlay a ui element or gameobject on the detected object bounding box. I’ve searched and can’t seem to find a solution. I know this here creates the bounding boxImgproc.rectangle(frame, new Point(left, top), new Point(right, bottom), new Scalar(0, 255, 0, 255), 2); Is there a way to grab those point and convert them into a Vector3?

I attached an image with the desired result.

6743221--777253--ButtonOverlay.png

2569

Thank you very much for reporting.
The cause of this bug is currently under investigation.

2570

I think the attached file will help you.
This example shows how to overlay a 3D object on the center of a detected face.
6746134--777703--DetectFace2DTo3DExample2.4.2.PNG

6746134–777706–DetectFace2DTo3DExample2.4.2.unitypackage (6.21 KB)

1 Like
2571

Thank you!! That worked exactly how I needed it to.

2572

Hi,
Can I use GStreamer for backend of VideoCapture class on Windows?
I installed GStreamer 1.18.3. But the following code does not work well.

VideoCapture capture;
capture.open("videotestsrc ! videoconvert ! appsink");
2573

In order to use Gstreamer, you need to enable Gstreamer and recompile OpenCV.
I’ve custom built a gstreamer-enabled OpenCV before and combined it with OpenCVForUnity.
For more details, see the section on “How to use OpenCV Dynamic Link Library with customized build settings” in ReadMe.pdf.

I have successfully connected the rtsp protocol by following the steps below.

  1. Follow the instructions on the following page to build OpenCV dynamic library.
    Compile Opencv with GStreamer for Visual Studio 2019 on windows 10 with and contribution modules
    OpenCV4.5.0
    GitHub - opencv/opencv at 78476a0fe25b761dfda813f9ba030e5d7f1aa82f
    GitHub - opencv/opencv_contrib at 37315babf9984a5b6aa981644a20dd8df1a5ff02

    6766333--781753--cmake.PNG
    6766333--781753--cmake.PNG759×962 43.7 KB

  2. Copy “OpenCVForUnity\Extra\dll_version\Windows\” to
    “OpenCVForUnity\Plugins\Windows\”.

  3. Copy the built OpenCV dynamic library to the "OpenCVForUnity\Plugins\Windows" folder.

    6766333--781756--dll_importsettings.PNG
    6766333--781756--dll_importsettings.PNG1057×551 79.6 KB

  4. Change to the attached VideoCaptureExample.cs.

capture.open(“videotestsrc ! videoconvert ! appsink”, Videoio.CAP_GSTREAMER);
6766333--781762--videotestsrc ! videoconvert ! appsink.PNG
capture.open(“rtspsrc location=rtsp://91.191.213.49:554/live_mpeg4.sdp latency=0 dulation=-1 ! decodebin ! videoconvert ! appsink”, Videoio.CAP_GSTREAMER);
6766333--781759--rtspsrc.PNG

6766333–781765–VideoCaptureExample.cs (11.4 KB)

2574

Hi, i’m testing the aruco example scenes but i am having a problem. When i use the dummy camera matrix everything goes well, but when i try to calibrate it with the calibration example scene, and use the new camera matrix and distortion coefficients in the webcam aruco example scene, it draws well the corners and pose (so i guess i did a good calibration), but something is off, since it offsets the ar cube, as you can see in the following images:

6766759--781885--upload_2021-1-26_12-28-1.png
6766759--781885--upload_2021-1-26_12-28-1.png1212×686 195 KB
6766759--781888--upload_2021-1-26_12-28-7.png
6766759--781888--upload_2021-1-26_12-28-7.png1206×673 169 KB

I tried calibrating multiples times and i always get the same result. Does someone know what could be happening?
I’m just using the example scenes without any modification and the last OpenCVForUnity with a logitech c920 hd pro, which has very little distortion. I also noted that this horizontal error offset of the cube position is proportionally similar to the difference between the default cx component of the camera matrix and the calibrated cx component of the matrix.

2575

Hi,

I’m working in a project: I have a picture, inside this picture I know N coplanar points (I have the world 3d coordinates and the coordinates in the image). I want to find out the camera position of the camera for the image. I managed to get it to work for python, but in your plugin I’m getting wrong output values for the same input data:

Python code:

print(f'FIELD objectPoints {world_points}\nimagePoints {screen_points}\ncamMatrix {camera_matrix}\ndistCoeffs {dist_coeffs}');
            _ret, rvec, tvec = cv.solvePnP(world_points, screen_points, camera_matrix, dist_coeffs, cv.SOLVEPNP_IPPE)
            print(f'FIELD rvec {rvec}\ntvec {tvec}');

C# code:

Debug.Log($"FIELD objectPoints {objectPoints.dump()}\nimagePoints {imagePoints.dump()}\ncamMatrix {camMatrix.dump()}\ndistCoeffs {distCoeffs.dump()}");
            Calib3d.solvePnP(objectPoints, imagePoints, camMatrix, distCoeffs, rvec, tvec, true, Calib3d.SOLVEPNP_IPPE);
            Debug.Log($"FIELD rvec {rvec.dump()}\ntvec {tvec.dump()}");

Python output:

FIELD objectPoints [[ 52.5    0.    34.  ]
[ 52.5    0.    -3.66]
[ 36.     0.   -20.16]
[ 41.5    0.     0.  ]]
imagePoints [[ 471.35842368  338.10150168]
[1740.883776    567.4959036 ]
[1561.767552    866.02291884]
[ 936.43209408  581.065272  ]]
camMatrix [[6.28360766e+03 0.00000000e+00 9.60000000e+02]
[0.00000000e+00 6.28360766e+03 5.40000000e+02]
[0.00000000e+00 0.00000000e+00 1.00000000e+00]]
distCoeffs [[0.]
[0.]
[0.]
[0.]]
FIELD rvec [[ 0.13082996]
[-0.57601962]
[-0.02956267]]
tvec [[-34.96040194]
[  3.35909129]
[ 62.97022695]]

C# output:

FIELD objectPoints [52.5, 0, 34;
52.5, 0, -3.6600001;
36, 0, -20.16;
41.5, 0, 0]
imagePoints [471.35843, 338.1015;
1740.8838, 567.49591;
1561.7676, 866.02295;
936.43207, 581.06531]
camMatrix [6283.60791015625, 0, 960;
0, 6283.60791015625, 540;
0, 0, 1]
distCoeffs [0;
0;
0;
0]
FIELD rvec [0.1538111422233873;
-0.6324571644188874;
-0.02627024069205628]
tvec [-32.95994179269246;
3.826083621349714;
72.16430511224465]

Any idea why?

2576

Thank you very much for the detailed instructions.

I followed the instructions and was able to display video in case of videotestsrc. However, in the case of rtspsrc, it does not work well. Only the first retrieve() works and the image is displayed, but after that retrieve() does not work at all.

I try using the same GStreamer command sequence for my C++ program using the same DLLs on the same PC, and there is no problem, the streaming video is displayed continuously.

What should I do?

2577

There are three types of camera parameters that can be obtained by OpenCV calibration: internal parameters (camera matrix), distortion parameters, and external parameters.
To get a perfect fit between the AR object and the camera image, you will probably need to set distortion parameters for the Unity Camera. You will probably need to use a shader for this.

2578

Thank you very much for reporting.

Is OpenCV version the same in Python and C# Code?
OpenCVForUnity uses OpenCV 4.5.0.
https://github.com/opencv/opencv/tree/78476a0fe25b761dfda813f9ba030e5d7f1aa82f
https://github.com/opencv/opencv_contrib/tree/37315babf9984a5b6aa981644a20dd8df1a5ff02

2579

The previous code was wrong. Could you please change it to the following?

capture.open("rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov", Videoio.CAP_GSTREAMER);

            capture.grab();
            if (capture.retrieve(rgbMat))
            {

6782525--785180--bunney.PNG

6782525–785186–VideoCaptureExample.cs (11.5 KB)

2580

This is the result of a similar test code running on C++.

c++ code:

#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/calib3d.hpp>

using namespace std;
using namespace cv;

int main(void)
{
    vector<Point3f> objectPoints;
    objectPoints.push_back(Point3d(52.5, 0, 34));
    objectPoints.push_back(Point3d(52.5, 0, -3.6600001));
    objectPoints.push_back(Point3d(36, 0, -20.16));
    objectPoints.push_back(Point3d(41.5, 0, 0));
    vector<Point2f> imagePoints;
    imagePoints.push_back(Point2d(471.35843, 338.1015));
    imagePoints.push_back(Point2d(1740.8838, 567.49591));
    imagePoints.push_back(Point2d(1561.7676, 866.02295));
    imagePoints.push_back(Point2d(936.43207, 581.06531));
    Mat camMatrix = (cv::Mat_<double>(3, 3) << 6283.60791015625, 0, 960,
        0, 6283.60791015625, 540,
        0, 0, 1);
    Mat distCoeffs = (cv::Mat_<double>(4, 1) << 0, 0, 0, 0);

    Mat rvec, tvec;
    solvePnP(objectPoints, imagePoints, camMatrix, distCoeffs, rvec, tvec, true, SOLVEPNP_IPPE);

    cout << "objectPoints : " << objectPoints << endl;
    cout << "imagePoints : " << imagePoints << endl;
    cout << "camMatrix : " << camMatrix << endl;
    cout << "distCoeffs : " << distCoeffs << endl;
    cout << "rvec : " << rvec << endl;
    cout << "tvec : " << tvec << endl;

    cv::waitKey(0);
}

c++ output:

objectPoints: [52.5, 0, 34;
    52.5, 0, -3.6600001;
    36, 0, -20.16;
    41.5, 0, 0]
    imagePoints : [471.35843, 338.1015;
    1740.8838, 567.49591;
    1561.7676, 866.02295;
    936.43207, 581.06531]
        camMatrix : [6283.60791015625, 0, 960;
    0, 6283.60791015625, 540;
    0, 0, 1]
        distCoeffs : [0;
    0;
    0;
    0]
        rvec : [0.1538111422233873;
    -0.6324571644188874;
    -0.02627024069205628]
        tvec : [-32.95994179269246;
    3.826083621349714;
    72.16430511224465]

I think the difference between C++ / C# (OpenCVForUnity) and python results is a python binding issue.