Computer Vision and Machine Learning integration.

Unity Engine
, ,
1

Hello guys i come from python background and,

I was getting familiar with ARFoundation and was thinking is there any possibility to do some image processing via python which can further assist in enhancing out AR experiences.

As per my current knowledge there is no proper way to integrate powerful python functionalities (OpenCV , tensorflow) to unity editor directly i have scraped the internet and 90% of examples rely on WebSockets for communicating between these two environments.

My Idea was to basically somehow send the frames of camera from unity to my python code via some sort of messaging queue service (such as rabbitmq) and then process the received frames in python for further image processing tasks.

However i feel sketchy about this approach as it looks resources expensive + requires nice internet connection if implemented for mobile … not to mention of uncertainty of will this ever work or not.

I have heard about ML agents unless my understanding is wrong ,they dont allow that much customization of code in python side for the work.

Can anyone would suggest some improvement in my approach or I would be really grateful hear other approaches.

Thank You !!

1 Like
2

Hi kislotaooankit,

I am also looking for a similar solution although my use case is a little different.

I am trying to detect body landmarks using openCV and use those to control character inside Unity.

Please share any resource if you have found anything. Likewise I’ll also share if I find anything.

Thanks!

3

So I just deleted the dll in ARcore and it worked, but this is no solution but a clumpsy work around.
Deleting the Google plugin in Barracuda did not work bc it reappeared all the time.
to be honest I still dont know if it will lead to some problems later on but I dont get ompiler errors anymore.

1 Like