Thankyou once again to everyone for their incredibly helpful contributions (Thomas, martejpad, inder2, mcelroy-jon, iseta and my apologies to anyone I may have have missed out). Here is a brief update including some notes that will hopefully prove useful to someone else taking the same journey. I also have soem questions that someone may be able to shine some light upon. If so I would be extremely grateful.
- Correct OpenCV SDK Python build_sdk.py script build parameters.
- Use of OpenCV functions in Android e.g. (VideoCapture, imread, selectROI)
- Input Unity VideoPlayer frames into OpenCV as Texture2D. Can’t get this to work or any other form of video input to OpenCV in android.
I have made some progress over the Christmas holidays. I have managed to utilise a lot of OpenCV functionality via the Windows plugin and I have had slightly more limited success using the Android equivalent despite martejpad’s excellent notes that got a brief shout out in a Thomas Hallberg video at 3:13:
Here is my configuration:
OpenCV: 4.5.0
Unity: 2020.1.7f1
Android Studio: 4.1.1
(1) I have gone down the route of using the OpenCV 4.5.0 sdk out of the box and I haven’t built the libraries from source or included the open_contrib content. I had a brief play with this going down both the CMake and python script routes. I wasn’t entirely sure of the correct parameters for the Git clone GitHub\opencv\platforms\android\build_sdk.py python script. If anyone knows these I would appreciate any input. I did use --ndk_path and --sdk_path
I had some trouble with getting Android Studio to build the native-lib so library reliably and found the following video very useful indeed with an overview and some useful diagrams and tips.
I found the following two lines particularly useful:
set(OpenCV_DIR $ENV{OPENCV_ANDROID}/sdk/native/jni)
this sets up the OpenCV root directory used in conjunction with environment variables.
and
find_package(OpenCV REQUIRED)
really useful for automatically using the sdk to configure settings.
Here are the settings that I use more often than not for CMakeLists:
#opencv
set(CMAKE_BUILD_TYPE Release)
set(CMAKE_CXX_FLAGS "-std=c++11")
set(OpenCV_STATIC on)
set(OpenCV_DIR $ENV{OPENCV_ANDROID}/sdk/native/jni)
message(STATUS "OpenCV library status1:")
message(STATUS "OpenCV SubDir: ${OpenCV_CONFIG_SUBDIR}")
message(STATUS "CMAKE CURRENT_LIST_FILE: ${CMAKE_CURRENT_LIST_FILE}")
message(STATUS "OpenCV library status2:")
message(STATUS " version: ${OpenCV_VERSION}")
message(STATUS " libraries: ${OpenCV_LIBS}")
message(STATUS " include path: ${OpenCV_INCLUDE_DIRS}")
find_package(OpenCV REQUIRED)
# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.
add_library( # Sets the name of the library.
native-lib
# Sets the library as a shared library.
SHARED
# Provides a relative path to your source file(s).
opencv-utils.cpp
native-lib.cpp )
# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.
find_library( # Sets the name of the path variable.
log-lib
# Specifies the name of the NDK library that
# you want CMake to locate.
log )
#find_library(jnigraphics-lib jnigraphics)
# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.
target_link_libraries( # Specifies the target library.
native-lib
# Links the target library to the log library
# included in the NDK.
${OpenCV_LIBS}
# ${jnigraphics-lib}
${log-lib}
log )
(2) I found it very difficult to use any of the OpenCV library calls for accessing media very difficult such as
VideoCapture for videos and
imread for images &
selectROI for easy bounding rect UI selection of ROI
all 3 were very straightforward in windows.
If anyone has any information on this front I would appreciate it.
I found the best way to achieve input via camera or video is via the Unity library and passing the data through to OpenCV via the methods largely explained above:
For camera use the following excellent video for setup
and then after you have your WebCamTexture backCam playing use the following with the SubmitFrame native C++ call. Code from the guys thanked above:
Color32[] pixels = backCam.GetPixels32();
GCHandle pixelHandle = GCHandle.Alloc(pixels, GCHandleType.Pinned);
IntPtr results = NativeLibAdapter.SubmitFrame(backCam.width, backCam.height,
pixelHandle.AddrOfPinnedObject());
int bufferSize = backCam.width * backCam.height * 4;
byte[] rawData = new byte[bufferSize];
if (results != IntPtr.Zero)
{
Marshal.Copy(results, rawData, 0, bufferSize);
outTexture.LoadRawTextureData(rawData);
outTexture.Apply();
}
background.texture = outTexture;
rawData = null;
pixelHandle.Free();
(3) For videos I haven’t manage to get it to work properly with Unity’s VideoPlayer. I changed mp4 format to enconding with H-265 without audio using VLC and imported the file to streamingassets in Unity to enable playing on an android device. The VideoPlayer input is rendered to a render texture and displayed on a RawImage within Unity. I have tried the following:
dest = new Texture2D(1280, 720, TextureFormat.RGBA32, false);
videoPlayer.Play();
/////////
dest.Apply(false);
Graphics.CopyTexture((videoPlayer.texture as RenderTexture), dest);
//
Color32[] pixels = dest.GetPixels32();
and then used the webcam approach but I am not getting any output although the IntPtr results are populated. The output data appears blank. I have tested displaying the dest texture on a separate raw image and it is rendered correctly?
Android threshold screenshots:
