Machine Learning Augmented Reality

Unity Engine
1

Hi guys.

We have a certain object that can be recognized easily using TensorFlow and Machine Learning. We wish to use this method for image recognition instead of Vuforia since that is not accurate enough. To do this we need a cusotm AR Engine that uses TensorFlow for the image recognition and uses custom code or an existing AR engine for the other activities like Pose Estimation and Unity Integration. Anyone has knowledge or experience of how to do this?

I can see that there are some TensorFlow libraries easily available for Unity, and we also have a trained graph that can detect our object easily. I am assuming for now this part should not be that hard The next step is to use the output from the TensorFlow to the AR Engine. This is where I see the maximum uncertainity. If I were to use an open source AR Engine(with modifications) to do the job, first I am not sure which is the right engine, and secondly how exactly I can go about doing all the steps and how easy it is to sepearate the modules of Image recognition and Pose estimation in the engine.

Any inputs are appreciated

Thanks

1 Like
2

Hi rahulkj
Sorry for this because my question is not answering the yours but I am integrating both TensorFlow with Unity and like you have experience about this topic Could you help me about to start in this topic? Thanks for your time
Alejandro

3

Hi Castana

Thanks for the reply. Looks like we both may be working on something similar if not the same thing. Let us get in and help each other out. Please check your private conversations. i will send you a message

4

Not specific to your question, but I wanted to make you aware of Unity ML-Agents GitHub - Unity-Technologies/ml-agents: The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.

5

Hi guys,
I am working on something similar. I use Tensorflow API for object detection and use TensorflowSharp to combine it with Unity and it works fine. But I want to put some AR stuff during the detection like if it detects an item, then a virtual item shows and overlays on the real one. I tried Vuforia but still have some problems. Do you have any suggestion of how to do the AR part or the pose estimation part?
Thanks!

6

Is TensorFlow fast enough to do video recognition near realtime?

7

you can convert the tensorflow model to coreml format and then use the open source unity/coreml implementation.

All you would do is edit the input/output handling.

1 Like