Hey everybody!
I am hoping to implement pano image stitching at runtime in Unity using EmguCV plugin, targeting mobile. So far I am able to use the high level Stitcher API (Stitcher.stitch method), but when the source images are too large or too many of them, the process is very resources heavy.
What I’ve tried:
- I am basically using 22 images taken by mobile photo camera (resolution approx. 2220x1080 each)
- I successfully implemented stitching in a separate thread (using Unity Jobs system), but it is still too heavy / takes too long / is unstable. I am using OpenCL acceleration which helps in terms of speed but seems a bit unstable in terms of memory (occasional crashes on subsequent attempts).
- I thought i would do the stitching pairwise after taking every pair of images, instead of loading them all at once to the stitcher after they are all taken, but that introduces a known issue of left side of the intermediate result getting gradually blurred (as a result of consequnt blending), so the Stitcher is unable to stich more images anymore at some point.
- I am aware that in OpenCV i could pass optional ROIs along with the input images, which might reduce both processing workload and undesired blurring effect, but i could not make this work using EmguCV (there is no such overload of the Stitcher.Stitch method with optional ROIS). I tried to apply the ROIs to the input Mats before passing them to the Stich call, but that doesnt seem to work either (creating a Mat using IntPtr and ROI from an existing Mat only allows setting a single ROI, while in this scenario it makes sense to set 2 ROIs for each source Mat/Image).
- I also tried to take all the images first, then recursively separate them into pairs and stitch each pair individually and so on, which seems to work quite well (using adaptively reduced PanoConfidenceThreshold for each next iteration to take the blurring into account), but sadly it crashes after 3rd iteration (at that point i have 5 intermediate stitching results, made out of original 22 images).
I was trying to put the code of my parallel job attempt but it doesnt let me paste save the post edit for some reason:/ I might attach its as a file if necessary.
I am running out of ideas here. I am thinking to try the other Stitcher API using EstimateTransform and then ComposePano, as it seems to have an overload that allows me to pass some image masks along with the input images, but frankly i dont understand how should i specify these for them to be in the required format IArrayOfArrays?? They are not ROIs after all?
But still I am not sure how to make proper use of parallelization either way:/
Obviously I am lacking deeper insight into the computer vision algorithms and / or OpenCV / EmguCV architecture here…
My main questions thus are:
- Does it even make sense to try to accomplish runtime panorama stitching in on mobile in Unity in the first place? I assume that if native mobile apps can do it, then it should be possible.
- Does it make sense to try call EmguCV API inside a unity Job?? I suspect this might the main problem here.
- In case of positive answer any of previous 2 questions, how to optimize the stitching properly? Do I have to dive into low level stitching API to stitch the images one by one and then SOMEHOW(?) build a panoramic projection out of the result?
Any suggestions will be highly appreciated!