Apply camera as texture to spatial map

Unity Engine
1

I’d like to do apply the camera’s capture results to the spatial map captured by the SpatialMapping manager. Basically, this:

I’d like to do that in real-time. I send my mesh data to a server, where I then visualise it. I want to be able to just do that with textures. I saw the photo blending thing, and that seemed like it would be quite similar, but I’m not really sure how I’d get from that to this. Any ideas?

Thanks!

1 Like
2

Let me know when you figure it out.

3

I’ve been experimenting a little with this but not yet fully successfully…freezing updates and shrinking the mesh, and then using a projector from the Standard Assets > Effects to try to display camera images. Have used diffuse mobile shader for the mapping mesh which does pick up the projector colors fine - but haven’t figured out the math if I can reproject from the spot where the photo was taking - what I need to change in parameters or if I’m mistaken that it is possible to make it semi-realistic ( would expect distortion not sure how majro). Been trying to do so statically for one view/wall… but not sure yet if this can work or if need to find other way to combine the mesh and photos from the HoloLens…So similar boat. Was trying to work it out from this example which instead of stills uses a movie texture to project on a mesh: Projecting a movie texture - Questions & Answers - Unity Discussions

4

I don’t think he’s using a projector in the video, looks like using quads with images at camera angles and then fused or overlaid…so using the camera API to get the original image and then resizing the captured image on a quad to match the equivalent area that was captured in the photo and aligning it within the room view.

5

With the spatial Mesh there are no UV’s, so applying a texture or picture is not possible. A way around this is a user might be able to use a shader to calculate everything needed for them based on the spatial mesh, then apply a texture through that shaders calculations.

The technique is call Tri planar mapping.

Video Link

@BrandonFogerty can provide more information if needed.

6

You can reproject your camera image onto a mesh either by setting the mesh’s texture to the camera feed and setting the vertex UVs using a script or by reprojecting inside of the pixel shader.

The former technique can be extended towards the construction of a persistent texture atlas.
The latter technique has pixel-perfect accuracy, but can only ever wok for what is currently visible to the user’s webcam.

MeshReproject.cs

Attach this to the mesh you’d like to have reprojected UVs:

using UnityEngine;
public class MeshReproject : MonoBehaviour {
  public Camera ProjectionPerspective;

  public float width = 0.2f;
  public float height = 0.2f;

  Mesh mesh;
  Vector3[] vertices;
  Vector2[] uvs;

  void Start() {
    ProjectionPerspective.enabled = false;
    mesh = GetComponent<MeshFilter>().mesh;
    mesh.MarkDynamic();
  }

  void Update() {
    ProjectionPerspective.projectionMatrix = ManualProjectionMatrix(-width, width, -height, height, ProjectionPerspective.nearClipPlane, ProjectionPerspective.farClipPlane);

    vertices = mesh.vertices;
    uvs = new Vector2[vertices.Length];
    Vector2 ScreenPosition;
    for (int i = 0; i < uvs.Length; i++) {
      ScreenPosition = ProjectionPerspective.WorldToViewportPoint(transform.TransformPoint(vertices[i]));
      uvs[i].Set(ScreenPosition.x, ScreenPosition.y);
    }

    mesh.uv = uvs;
  }

  static Matrix4x4 ManualProjectionMatrix(float left, float right, float bottom, float top, float near, float far) {
    float x = 2.0F * near / (right - left);
    float y = 2.0F * near / (top - bottom);
    float a = (right + left) / (right - left);
    float b = (top + bottom) / (top - bottom);
    float c = -(far + near) / (far - near);
    float d = -(2.0F * far * near) / (far - near);
    float e = -1.0F;
    Matrix4x4 m = new Matrix4x4();
    m[0, 0] = x;
    m[0, 1] = 0;
    m[0, 2] = a;
    m[0, 3] = 0;
    m[1, 0] = 0;
    m[1, 1] = y;
    m[1, 2] = b;
    m[1, 3] = 0;
    m[2, 0] = 0;
    m[2, 1] = 0;
    m[2, 2] = c;
    m[2, 3] = d;
    m[3, 0] = 0;
    m[3, 1] = 0;
    m[3, 2] = e;
    m[3, 3] = 0;
    return m;
  }
}

GPU Reprojection

First set the Camera Projection Matrix variable in C#:

  void LateUpdate() {
    Shader.SetGlobalMatrix("_camProjection", FrontCam.projectionMatrix * FrontCam.worldToCameraMatrix);
  }

Then calculate the UVs inside of the mesh’s material’s shader:

            uniform float4x4 _camProjection;

            float2 WorldToViewport(float4x4 camVP, float3 worldPoint)
            {
                float2 result;
                result.x = camVP[0][0] * worldPoint.x + camVP[0][1] * worldPoint.y + camVP[0][2] * worldPoint.z + camVP[0][3];
                result.y = camVP[1][0] * worldPoint.x + camVP[1][1] * worldPoint.y + camVP[1][2] * worldPoint.z + camVP[1][3];
                float num = camVP[3][0] * worldPoint.x + camVP[3][1] * worldPoint.y + camVP[3][2] * worldPoint.z + camVP[3][3];
                num = 1.0 / num;
                result.x *= num;
                result.y *= num;

                result.x = (result.x * 0.5 + 0.5);
                result.y = (result.y * 0.5 + 0.5);

                return result;
            }

Once you have this, you can sample your texture using:

uv = WorldToViewport(_camProjection, i.worldPos);

(Make sure worldPos is being passed in through i)

7

I’m not sure how I’d map individual images to each mesh. I tried doing it on when a surface is received, to get the current image/texture of the camera, and apply it as such above, but it just gives me a pseudo-skybox view with some mildly warped meshes following the camera texture.

8

I managed to get the GPU Reprojection code shader working! Thank you! But yes, like you said, it only works for what the camera currently sees. It either repeats/clamps everything else until you move to it, causing the prior ones to lose their texture. I tried snapshotting it, and setting it that way, but that doesn’t really work either.