Deformable surface tracking

Unity Engine
,
1

Hello and happy new year!

I am looking for a way to perform deformable surface tracking, similar to what Disney do in this video when tracking pages of a book being turned or partially deformed. I have also seen some nice videos on linkedin recently where pillow surfaces can be tracked and materials applied to them at runtime.

Does anybody know of a tool that could be available to use in AR foundation or would have any ideas/knowledge on how to achieve that kind of tracking?

Many thanks

Michele

2

AR Foundation is a scripting interface that allows you to deploy cross-platform AR apps to Android, iOS, and OpenXR devices. Unity does not implement AR features-- we simply support a cross-platform interface to platform-level AR implementations.

Neither ARCore nor ARKit support deformable surface tracking, so you would need a bespoke solution for this use case. (I am not aware of one, but good luck!)

3

Thanks @andyb-unity !
Would it be possible to use the point cloud and something like the below to achieve the tracking?

        var mesh = new Mesh();
        // Set the vertices and triangles of the mesh from the point cloud data
        mesh.vertices = pointCloud.positions;
        mesh.triangles = GenerateTriangles(pointCloud.positions.Length);
        // Recalculate the normals of the mesh
        mesh.RecalculateNormals();

        return mesh;

Or is there anything in the meshing examples that could help? Sorry I haven’t had time to look at the code in there yet, I will do so asap.

Many thanks

4

If it were this simple to do Disney wouldn’t need an entire research group to achieve it.

5

@Michele_P_LF That code simply generates a mesh from a point cloud. It does not achieve image tracking. To be clear, there is nothing currently in ARCore, ARKit, or AR Foundation that will help with this use case.