I want to make an Augmented Reality application (using the Qualcomm QCAR plugin) in which rendered objects interact with real objects. The plan is to make a 3d model (no textures) of a room and then place a marker in the real room. The marker position relative to the real room is known. Now the 3d model of the room can be used for collision detection and culling between rendered objects and the real room.
But I am not sure how to approach this. I have to make the polygons or (non-existent) textures of the 3d model of the room transparent. Otherwise the 3d model of the room will be rendered on top of the camera image, and this should not be the case. If the polygons or textures of the 3d model of the room are transparent, I presume it will work fine for collision detection between the dynamic rendered objects (say a race car) and the real camera image. However, the dynamic 3d object in question will not be occluded by the 3d model of the room if it is partially or fully behind some geometry, because of the transparency.
How do I solved this? Playing with the Z-buffer? Making a special shader?
Any ideas?
Hi!
I think it will be difficult to create a general solution to this problem, since there is no way for you to calculate the depth of anything from the camera signal, apart from your markers. I guess you could work with several different markers to create an occlusion of sorts.
As for the collision detection, collisions are handled independently of shaders and materials, so they will work fine.
Actually, the depth depth information from the camera image IS available. Not from fancy infrared range finders or stereo images but from the 3d model itself. Imagine having a detailed 3d model from a room. It contains the floor, ceiling, walls, chairs, doors, etc. This 3d model is exactly the same as the real room, all the dimensions, and orientations of the 3d model line up exactly with the camera image, with the help of the image tracking software (QCAR). Now, as long as you don’t shift the chairs or move the doors, the 3d model and the real camera image should overlap perfectly. So the depth information of the camera is extracted from the 3d model of the room. Of course this is not the case with dynamic objects in the real world such as the cat or dog walking around, but it is an approximation which should be sufficient for gaming.
See this example of what I mean:
In that example you can actually move the geometry around (at 1:00) but that is not what I am after. A static environment (except for the rendered moving models) is enough.
Hi!
I am not sure what you are asking here really?
How would you have the depth information from the environment? Unless you know beforehand in what room the user will run the app, and can create a model of that particular room in advance. I see limited use of such a thing, but maybe I don´t understand what you are trying to do?
That is exactly how the app is going to work. The user creates a 3d model of a specific room. The environment to be used has to be known beforehand, and the app will only work in that environment. The 3d model creation is not done using a 3d modeler on the pc (it can be though) but with the phone itself. Sort of an AR model builder. Exactly like in the video I posted. The 3d model is created using the camera image. If the 3d model creation is user friendly and quick enough (like in the video, but using a smartphone with touchscreen instead of a PC with mouse) then I think the incredible user experience such an app can provide outweighs the effort the user has to put in to create the 3d model. And rooms don’t tend to change much, especially the walls, floor, and ceiling. So you only have to create the 3d model once.
I mean, imagine shooting zombies which break through the walls in your own living room. Or flying a model airplane around in your house, careful not to crash into the walls. Or driving model cars around on the table, falling down as you reach the edge. Isn’t this worth the trouble of creating the 3d model?
Anyway, I am sure users are willing to do this. So how do I handle the culling, like in that video?
I think the big obstacle will be exactly how to go about modeling your surroundings using the phone. This is going to be pretty difficult, both conceptually and technically.
Once that is done, everything else will be easy. You don´t have to worry about occlusion etc since that will be taken care of by 3D renderer.
Could be pretty cool, but I am pretty sure it will be a lot of work to create the “good-enough” modeler on the phone.
I think it won’t be easy too but that doesn’t stop me.
You say that the occlusions will be taken care of by the 3D renderer, but how? If a polygon is transparent (which it has to be, otherwise the room model paints right over the entire camera image), how is it going to occlude parts of model race car driving around as it moves behind some room geometry?
The qualcomm AR package contains a material called DepthMask.mat.
This material is transparent but does write to the z-buffer, it performs the occluding you need.
It is very interesting indeed. I had a play with it and it does work. It needs some more attention though before it is noob-proof but at least it is open source and free. I don’t think I will use that system on android though. I think it will be a source of a lot of RSI because of the small screen. Using a mouse is probably much easier. So I think the user will have to create the 3d model of the room on the PC.
What might work on the android is using a simple primitive (cube, cylinder) creation tool with scaling and translation. No rotation, easier to just physically align the marker with the real table or whatever the geometry is. This will do fine for simple geometry, even in video mode instead of static-photo mode.
I took Insight3d for a testdrive too.
It crashed a couple of times because I used too many photo’s.
I had hoped that it would create geometry automatically, but you have to create polygons between the generated vertices yourself.
This proved too labor intensive, as I used a test object that was waaayyyyy too curvy
Another test, some other day…
Yeah, automatic geometry creation would be nice. But I think the only thing which comes close to that is something like Canoma. With that you have to use 3d primitives like cubes and cylinders to create the geometry. This is way faster than creating separate polygons but you cannot make complex curved objects with it.
It is a pity that the author of Insight3d doesn’t reply to his email. I posted several bug reports and feature requests but he is either on holiday or completely abandoned the project.
Another option is to use Autodesk ImageModeler. I haven’t tried it but the youtube video’s look promising.
EDIT: I had a go at ImageModeler but it is not really suitable for doing indoor scenes. The photos need to be taken with a wide angle lens, really. And they need to be stitched together (panorama) before use which requires yet another software package. So now I am looking at Autodesk Project Photofly http://labs.autodesk.com/technologies/photofly/overview/ http://labs.autodesk.com/utilities/photo_scene_editor/
It handles image calibration automatically so that saves a lot of work. And it is free. The downside is that you have to upload your pictures to a server to be processed.
I made a 3D model of the fort itself and then as you suggest used a custom shader on the model that doesn’t render it but still occludes the virtual objects.
I have had good success with making 3D models from the Autodesk 123D - http://labs.autodesk.com/utilities/photo_scene_editor/ - and if you have a static room you should only need to make the model once, so the uploading of your photos to cloud shouldn’t have too much of an impact on your project… now if you need to be able to change the geometry of the room model on the fly (opening closing doors, moving chairs around, etc.) that is a whole different ball-game!
There may be some scope for looking at Kinect integration which will provide the live depth-mapping for occlusion at least close-up…
Cool game that is!
I am glad to see that Autodesk released that photofly app for free. It is still not very suitable to create planar geometry with, but for organic shapes it is great.
