Hands over 3D object - Vuforia AR - Unity

Hi all,
I’m going crazy searching to make this thing. I want to show my hand over the 3D object fixed with Vuforia over a marker.
So, Vuforia handles the video and put the object with the marker reference. This is well documented and easy to do.

But what I can’t do is the part to take the camera image (without 3D object) and put it over the scene. Then, I have a shader that does the chromakey work and delete the part with a particular color, so to create an hole and leave only the hand over the 3D object.

In this thread, a user said that Vuforia let to take the camera image (without augmentation) as a texture. So it is possible to take the image and put it over the camera and then modify it with shaders. But how??

I hope that someone can help me. How can I take the camera image (without 3D object) and put over all?

Thanks in advance!

I have finally done with the help of an other user.
Here you can see how can be done:


I try to do the same thing and I think I’m pretty close, for what I have learn and ignore till now (ignorance is like an invisible wall, you see the other side but you cannot find the door to go there, especially where there is no door).

I haven’ t accomplished it yet, cause i am not that good programmer but I have ideas to share, and I am pretty welcome to code solutions and development disquation…

What I think, is pretty close to the video with the “holding wooden cube” and an instance of a step furthermore…

In the video they use the green screen trick (alpha removal / chroma key). That is why the green fabric.

Both background and real cube are in colour tones of green.

In the unity setup inside the ARCamera prefab there is a plane (AARCamera → Camera → plane) that renders the webcam / smartphone camera input video (what you see of the real world, in your screen, is a simple video projection). What we can do is repeat this video, before the augmentation repeat this video with the alpha channel removed. What it’s left is the hand.

Possible problems to take under consideration:

  1. Scale problems. The hand might be bigger. ↔ Maybe ortho camera is a solution.

  2. Mark detection and alpha removal. There is a thin line between doing everything in tones of green and computer vision reads clear the mark identifier. ↔ This is a try-fail-repeat procedure. What I think is creating / using a VUMARK in two green tones and check it in the vuforia target manager. What tones of green can be “deleted” needs further research. But if you achieve create the right VUMARK then you have a AR logo to print on stickers and stick them everywhere!

  3. In real everyday use we cannot have a chroma key fabric with us, so to play. This is the real hard part. ↔ What I am thinking, but still no results (not that I have done the above, just being pretty exited) is a hard code mixture of stencil shaders and chroma key removal. Somehow to repeat part of the background video input. The part that is being hidden by the augmentation (fingers / hands) with part of the real background that hopefully is chroma key masked by VUMARK. I haven’t think any solution for the part of hand that cannot be masked by the VUMARK but I 'll come to a solution. Maybe skin recognition colour algorithms, but I have no idea if this do exists (just read it in a post). I also have found no tutorial / information / prefab and instructions on the use of stencil shaders.

  4. All the above maybe easily run on the pc / unity editor, but it is not sure that our options will run on a mobile device. ex.: I have read somewhere that you maybe have to manually set smartphone to 24bit buffer something, for stencil shader to work, etc.

That’s all till now,

Any ideas or implementations, even small parts of a single solution, please share…