ARToolkit extension

Unity Engine
1

Hi There!

I’m quite interested in ARToolkit ( http://www.hitl.washington.edu/artoolkit/ ), a software library for building Augmented Reality (AR) applications. These are applications that involve the overlay of virtual imagery on the real world.
GameBlender and Virtools already have plugins to communicate with ARToolkit. Check http://www.ash.webstranka.info/?p=36 for an example movie.
My question is, how difficult will it be to combine ARToolkit with Unity? And could it be possible to make a connection without owning Unity pro? I am not a programmer by any means. I actually just started trying Unity to learn programming in a more playful way.
If anyone is interested in making a connection, ARToolkit plus ( http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php ) would be an even better base to build ontop.

Links:

GameBlender plugin: http://www.ash.webstranka.info/?p=45

Virtools plugin: http://mirevicast.medien.fh-duesseldorf.de/projects/virtools-extensions

As an example what kind of games you could create, check this amazing project by Julian Oliver: http://julianoliver.com/levelhead

Greetings,

Wim

2

From an initial scan of the docs it seems like its written in C and i couldn’t find any API in C# so think it will have to be done in a plugin but it should be possible. It might be hairy though to actually combine the rendering pipeline into unity, but can’t say for sure as i havent done it :smile:

Is a cool technology.

Regards,
Marc

3

Theres a c# version of NyArToolkit, im sure this could be used… i dont have time to test it out, but please someone do, id love this as well (indie)

http://nyatla.jp/nyartoolkit/wiki/index.php?FrontPage.en

4

Hi,

I’m also doing a plugin with ARToolKit.

A few things I managed to do :

  • The pattern recognition works.

  • I’ve integrated the image grabbed by the webcam on a plane (Texture2D) which is far away from the Unity camera.

  • Thanks to openGL I have access to 2 matrices [16] of 16 elements that I convert in 2 matrices [3][4] so i can extract easily a position vector and a quaternion with an ARToolkit function “arUtilMat2QuatPos”. The first one is the projection matrix for the Unity Camera (the real webcam is not in the position (0,0,0) and rotation (0,0,0) in Unity world, this is due to camera calibration). → “glGetFloatv(GL_PROJECTION_MATRIX,MatrixProjection);”
    The other one is the model-view matrix for the Unity object which will appear on the pattern. -->“glGetFloatv(GL_MODELVIEW_MATRIX,MatrixModelView);”

-The object (in Unity → a cube) I want to be on the pattern follow the pattern when I move the pattern.

Nevertheless, I have some projection (or position) problems : the object is not on the middle of the pattern but is shifted.

I think that the problem is that one or two of my gameObject (the camera, the cube and the plane) is not is the right position. Because when the pattern is on the left extremety of the screen, the cube has a better position than if the pattern was on the right extremity of the screen.

The distance plane-camera is a constant and is set to an arbitrary value (“4150” so that the camera look only at the plane in the Unity world)
The plane move only one time with the camera in the “start” function (due to the projection matrix).
The cube move alone (all the time) (due to the model-view matrix) in the “update” function.

Does someone (who knows well Unity/OpenGL or ARToolkit) thinks that I’ve missed a step to determine the positions of my gameObjects.

Or maybe the problem is somewhere else :s .

167620--6041--$unity_176.jpg
167620--6041--$unity_176.jpg900×570 24.7 KB

5

I’ve likewise implemented an ARTk plugin and got exactly to where solmyr is. Something gets mucked up in the translation from the plugin coordinates to Unity. I tried asking on the ARTk forums and emailed the maintainer about it and all I ever got was a rather brusque reply that I must be reversing the projection matrix (ie, calculating it in camera space rather than world space, which can be, and was, tested by adding a single function call to the plugin). There really isn’t much support for ARTk anymore outside of the rather lackluster forums, unfortunately, and the maintainers didn’t seem interested in looking into the problem at all.

6

yeah… it’s really strange, it’s working perfectly when the drawing are done in OpenGL (in ARToolKIt code samples).
I’ve followed every function that is necessary in their code to get the 2 matrices. Then the adaptation to unity should have be easy :?

7

looks interesting, is this useable for indie too ?
if so, could you make an guide or example project ?
i’d like to try it out

8

It’s Pro only. When last I checked the ARTk library is C and C++ only (with wrappers for some other languages), which means it has to be done via plugin.

9

This is a long shot, but:

as far as I can tell, Unity uses a left handed co-ordinate system. OpenGL uses a right handed co-ordinate system.

If you are stacking your matrixes in Unity, do them in the reverse order. Let me know if this works.

10

I can’t speak for solmyr, but I wasn’t mucking with the rendering pipeline at all (aside from using Graphics.DrawTexture() to overlay the incoming video image) since doing so would require getting the camera modelview matrix post-render, sending it to the plugin and drawing the AR objects there and then setting it again (which defeats the purpose of using a 3D engine). I was simply using the transformation values ARTk produced to update the position of Unity prefabs. There is a utility function in ARTk to get a right-handed transformation matrix, but just from looking at my notes (the project won’t open in 2.5 without some conversion I can’t really do right now), we were actually getting different transform values in the trace from the plugin versus what Unity was actually receiving.

11

Yes I’m doing it in the same way (don’t work either)

I don’t need it, i’m just switch Y and Z axes. Apparently it works because the cube is following the pattern in all directions (up/down/left/right/front/back)

I’d rather think that the problem comes from an openGL transformation we don’t understand (with the 2 matrices projection/model-view)

Or perhaps it comes from another coordinates problem:
-the projection matrix has to be inversed (because we need camera coordinates in the real world)
-the modelview matrix musn’t be inversed (because it’s the pattern coordinates in the camera coordinates)

3 possibilities , 3 failures ^^

167955--6056--$comparaison_151.jpg
167955--6056--$comparaison_151.jpg576×600 28.5 KB

12

Another strange thing (burnumd have the same problem I suppose)

With ARToolKit code sample, the openGL modelview matrix is :
-0.210773 | 0.411159 | -0.886861 | 0
0.969434 | -0.0286372 | -0.243674 | 0
-0.125586 | -0.911114 | -0.392556 | 0
51.7304 | -38.3579 | 384.759 | 1

With my code in Unity3d, the openGL modelview matrix is :
0.02105 | 0.377307 | 0.925849 | 0
0.99863 | 0.0364404 | -0.0375552 | 0
-0.0479081 | 0.925371 | -0.376023 | 0
51.9647 | 21.9308 | 391.948 | 1

The pattern is exactly in the same position.

The more important thing is the position of the pattern ( i’ll deal with the rotation problem after). The position XYZ is the last line:
51.7304 | -38.3579 | 384.759 in the first case
51.9647 | 21.9308 | 391.948 in the other case.

:shock:

13

That’s pretty much exactly the problem we were having. We tried to do some corrections to the values, but it didn’t seem to correspond linearly and I, at least, couldn’t see a pattern to the deviation. I didn’t use the modelview matrix directly (I used the utility function to convert the matrix to two double arrays. Looking at your data, I wonder now if it’s a bug in the plugin system. Maybe an overflow or type conversion issue since ARTk gives the matricies as doubles rather than floats? From looking at my plugin, it doesn’t seem like it would be the latter, since my plugin print statements display the converted float value and I saw the same issue. It didn’t occur to me at the time to report it because I assumed it to be either an ARTk problem or something I was just doing wrong, but if you’ve got a working(-ish) project, you might consider submitting a ticket to Unity QA with your project (along with the reams of fiducial patterns so they can actually test it).

14

just well… did you try mirroring the camera texture >_> ?

15

First, thanks for your help :).

Can’t be that, I send a floats matrix to Unity.

The main steps of my program are (in red C# script , in blue C++ plugin , green comments):

-Start()
–>initARToolKit(); // init the webcam and the projection matrix
–>GetQuatAndPos(quaternionCam, positionCam, MODE_CAM,INVERSE_CAM); // get position and rotation for the cam using arUtilMat2QuatPos(newMatrix, q, p);
// “quaternionCam” and" positionCam" are float matrices whereas “q” and “p” are double matrices
→ UpdateCamera(…) // update positions and rotations of the Unity camera

-Update()
→ mainLoop(); // grab the image,detect the marker, calculate the “patt_trans” matrix
–>UpdateTexture1(…) //Update the texture on the plane like in the texture plugin
→ if(IsPatternDetected()
){
---->GetQuatAndPos(quaternionOrient, positionPatt,MODE_CUBE,NULL); // get the position and rotation of the cube in float matrices
---->UpdateCube(…) // update positions and rotations of the cube
}

Here the code for “GetQuatAndPos(quaternionOrient, positionPatt,MODE_CUBE,NULL)” . You can see that there is no problem of double or float conversion through the unity plugin

{
argConvGlpara(patt_trans, gl_para);
glMatrixMode(GL_MODELVIEW);
glLoadMatrixd( gl_para );
glTranslatef( 0.0, 0.0, 25.0 ); 
glGetFloatv(GL_MODELVIEW_MATRIX,matriceModelView);
for(int ii = 0; ii < 3; ii++ ){
	for( int jj = 0; jj < 4; jj++){
            newMatrix[ii][jj] = matriceModelView[jj*4+ii]; 
	}
}	
					arUtilMat2QuatPos(newMatrix, q, p);

for( int ii = 0; ii < 4; ii++ ){quat[ii] = (float) q[ii];}
for( int ii = 0; ii < 3; ii++ ){pos[ii] = (float) p[ii];}
argSwapBuffers();
}

Already did because the camera grab a mirrored image (and when I move my pattern in the left side in reality, it does the same thing in the texture).
Moreover the problem isn’t here zeroZshadow.

There is nothing that could explain the difference between the values in the modelview matrices in the 2 cases (ArTk sample of code and my unity plugin) with a pattern in a same position.
-I follow every step they do
-I don’t modify the intermediate matrices
-I compare the openGL modelview matrix that is used in the display function to my openGL modelview matrix that is used just before sending the coordinates to Unity.

16

Hi, I didn’t read this very closely, but I am incorporating StbES into Unity and have change both the position and quaternions to Unity’s left handed system. For the position of a marker (z up) I found I had to invert the Y value. For the quaternion, I had to invert the X and Z quat values. We are not putting the projection matrix delivered by StbES directly into the camera because it seems to interfere with picking. So we are just setting the transform and FOV from calibration data.

17

Hey Guys!

This is my way to calculate the position and rotation of an object using ArToolkit. It works fine if the camera position is (0,0,0) and looks to +z axis, the image plane is at position (0,0,960) with a scale of (128,1,96):

// position
double[] trans = new double[3];

// rotation vector as quaternion
double[] rot = new double[4];

// Get the translation and the rotation with
// the AR function arUtilMat2QuatPos
getDetectedTranslationRotation(trans, rot);

// Set position
transform.position = new Vector3(-(float)trans[0] * 1.5f, -(float)trans[1] * 1.5f, (float)trans[2]);

// Set rotation
transform.rotation = new Quaternion((float)rot[0], (float)rot[1], -(float)rot[2], (float)rot[3]);

The getDetectedTranslationRotation is a DLL function where patt_trans is the dedected marker matrix:

EXPORT void getDetectedTranslationRotation(double* translation, double* rotation){
	arUtilMat2QuatPos(patt_trans, rotation, translation);
}

This implementation works fine in my case, even if the marker are in the corner of the webcam image.

177152--6332--$unity_ar_254.jpg

18

Can someone publish sources? :slight_smile:

19

wow, the sources would be awesome: :shock:

20

Hi,

I have experience in the AR side of things - I can code in C/C++ (amongst other things) and understand camera matrices.

But I’m new to Unity3D. What do I need to do to get webcam video streaming into a Unity 3D authored demo?

Will I need the pro version of Unity3D? Or will the indie version do it?

Carl