How to implement Cached Spatial Mapping on the device with Unity?

Unity Engine
1

So, I wanted to use the cached spatial mapping route instead of continuous ( https://developer.microsoft.com/en-us/windows/holographic/room_scan_ux#Cached_versus_Continuous_Spatial_Mapping ).
You see it in a game like Fragments – first step is to sufficiently map a room. On subsequent loads, the same room is loaded. You can select from a previously mapped space.

I was trying to implement this with Unity, but it doesn’t seem to be supported. I’ve downloaded Microsoft’s HoloToolkit for Unity and sniffed around all of the Spatial classes. The FileSurfaceObserver is only used by the Unity Editor. The SpatialMappingObserver is used on the device, but does not support loading from a file. It’s a continuous observer, but can be turned off.

I can save the mesh from Spatial Mapping Observer to disk using the MeshSaver class, but actually I noticed it is about to be deprecated along with some other related classes I was using ( Deprecate remote spatial mapping classes · Issue #39 · microsoft/MixedRealityToolkit-Unity · GitHub ).

How can I load a mesh from disk, and have the hololens orient the user within that space (preferably with events telling me when it can’t match the space)?

2

The typical way to achieve what you’re looking for is to map the room normally and then pause or halt updates so the data is frozen. The HoloToolkit SpatialMappingRenderer and SpatialMappingCollider both support this through their FreezeMeshUpdates properties. If you’re writing the script for spatial mapping yourself, it’s as simple as not calling SurfaceObserver::Update once you have data you’re happy with. I think this is what Microsoft is saying in the link you posted.

As far as I’m aware there isn’t a channel to tell the HoloLens about geometry in the room and have it understand that spatially. There shouldn’t be a long wait to map the room the next time your application runs (aside from the cost of ingesting the spatial mapping data) since the device remembers its spatial data at the OS level between boots.

Let me know if I can clarify anything for you.

3

Okay thanks, I haven’t tried the SpatialMappingRenderer but I will try that route out instead.

How does the Fragments game do it? It doesn’t seem to have any re-mapping step on app relaunch, and seems to recognize all of the previously scanned features. The mapping step takes a while in that game… Additionally, you can load

Is it likely that they are using a custom solution? I can imagine building a cleaned up, optimized scan of the full room, performing feature detection on it, and then comparing it to smaller, on-the-fly spatial meshes from the hololens… but that is very much non-trivial.

4

One more thing – Could you clarify the interaction of SpatialMappingRenderer, SpatialMappingManager and SpatialMappingObserver?

5

I don’t know how Fragments does its spatial mapping. My best guess is that it’s scanning when the app starts, doing some processing to detect planes and play spaces, and then potentially saving/loading those through the persistence API. If Fragments is built in Unity, I would be surprised if it used a custom spatial mapping solution.

I’m familiar with the SpatialMappingRenderer, SpatialMappingCollider, and their shared parent SMBaseAbstract. These are components that Microsoft built to automate common spatial mapping use cases. SpatialMappingObserver seems to be a different component built by Microsoft that does largely the same things. SpatialMappingManager seems to manage SpatialMappingObserver components but a casual inspection hasn’t revealed terribly much more.

Sorry for the late response.

6

Hey, was this figured out? The Fragments way of having you scan the room prior to playing the game works great. I wish they had that tutorial after the Holograms 230 one.

7

Hello,

It might be easier to ask questions about a Microsoft game on the Microsoft forums for Hololens development.

It help to explore the spatial mapping renderer and collider components in 5.6 or 5.5 builds of unity. Have you tried saving a scan using the world anchor store?

8

yep any update on this would be great!

9

There’s currently an open Pull Request that might cover this over on Github for the HoloToolkit.

It addresses saving meshes for use later.

10

Sorry, I might be too late for this conversation.

This is how I implemented a already scanned room into my app:

  • Scan the room you want to see in your app and save it from the device portal.
  • Copy the file to your project.
  • Setup your app to work with hololens and add the spatial mapping prefab to the hierarchy.
  • In the prefab, Under the spatial mapping manager component uncheck auto start observer.
  • Now, drag your mesh data from the project folder and drop it in room model field of the object surface observer.
  • Open the object surface observer and change the following line from

#if UNITY_2017_2_OR_NEWER
if (!UnityEngine.XR.XRDevice.isPresent && Application.isEditor)
#else

  • To

#if UNITY_2017_2_OR_NEWER
if (UnityEngine.XR.XRDevice.isPresent || Application.isEditor)
#else

Hope I helped anyone.

2 Likes
11

Thanks Mr_Mystery !

I’ve setup a previously scanned room into my hololens project following your recomendations.
Could you please help a bit further ?
How do I achieve holograms (objects) placement within this room ?

Thanks in advance.

12

To be more precise… I would like to know if it is posible to place objects in a pre-scaned room from unity editor and get them back at their right position in hololens when in that saved room.

Hope that makes sense…

13

Hi !

I just received my hololens and i’m looking for a solution to do an application supposed to run in only one room !
So i get the spatial mapping from the web admin interface of hololens and i’m looking to know how i can fix holograms on a specific wall in unity.
When i start my application for the first time, i would like my hologram already in place !

(btw i apologies for my bad English skills :smile:)

14

Well, there is a world anchor component in the holotoolkit. All you have to do is add it to your gameObject.
But, Adding a gameObject with world anchor manager won’t allow you to move the gameObject. If you have any problem let me know.

15

I think that’s possible. Because the spatial mapping data now you use works as a gameObject getting instantiated.

But, I’ll give you a better solution to this.

16

I read a lot of thing about that. The problem i have is that if i put the object on the right of the origin, when i start my application it will be placed regarding the the place et where i’m looking.
here an exemple of what i want to do :

I scanned a room and get the .obj file. I want to use this file to fix my hologram in unity and then when I start my application, the hololens reconize the room (by comparing the .obj file and the live scanning and then my holograms appears for exemple on the wall i place the gameobject in my unity scene.

Right now, my game object is fixed in relation to the origin position of the hololens. And i would skip the “configuration start” in my application.
So take a scan, place my hologram in unity, deploy, start and voila my holograms are in place

Can i do that with world anchor ? Maybe i use them in a wrong way…

17

There is a better solution I wanted to say. We’ll, you can find it somewhere over the internet. The downloaded spatial data will act as a GameObject and will be placed right at World position keeping your head as it’s origin. Now, create a separate project and import the Holotoolkit and Examples packages and setup the scene.

  • Drag the spatial mapping prefab to the Hierarchy and leave it there. (No changes needed)
  • In your project panel search for Cursor.cs script and open it.
  • You’ll find on the line 397 a method like the one below:

Change it from:-

public virtual void OnInputClicked(InputClickedEventData eventData)
{
// Open input socket for other cool stuff…
}
To:-

public virtual void OnInputClicked(InputClickedEventData eventData)
{
if (SomeBool)
{
SpatialMappingManager.Instance.StopObserver ();
SomeBool = false;
}
}

  • As we all know, this method is used for air-tap gesture. So, you can use this method from the script to stop the spatial update by using a bool. And add using HoloToolkit.Unity.SpatialMapping; to get access to the Spatial mapping classes.
  • Now, you’ll have your own spatial data.

And you can use the same method to place your hologram in the spatial data you’ve created depending on your needs.

18

My project needs to run on 5 Hololenses so it would be great if I could position the objects within Unity based on the pre-scanned model, instead of finetuning the room for each Hololens using the WorldAnchors.

After searching a lot about how to do that, I think this thread is almost getting close to a working answer. But either I’m missing one instruction, or perhaps the approach is not the right way to achieve the above.

I’ve dragged the pre-scanned room model to the Object Surface Observer and I disabled the “auto start observer” option. But when I view the app on my Hololens, I see a pre-scanned mesh but it still depends on the origin and direction of the camera at launch.

What is the piece of script that puts everything in place? It will need some kind of room scanning to do that, matching the current room to the pre-scanned room. But if the observer is not active, perhaps that matching does not occur?

19

You can follow this.
The downloaded spatial data will act as a GameObject and will be placed right at World position keeping your head as it’s origin. Which definitely make your spatial data awkwardly placed. Now, create a separate project and import the Holotoolkit and Examples packages and setup the scene.

  • Drag the spatial mapping prefab to the Hierarchy and leave it there. (No changes needed)
  • In your project panel search for Cursor.cs script and open it.
  • You’ll find on the line 397 a method like the one below:

Change it from:-

public virtual void OnInputClicked(InputClickedEventData eventData)
{
// Open input socket for other cool stuff…
}
To:-

public virtual void OnInputClicked(InputClickedEventData eventData)
{
if (SomeBool)
{
SpatialMappingManager.Instance.StopObserver ();
SomeBool = false;
}
}

  • As we all know, this method is used for air-tap gesture. So, you can use this method from the script to stop the spatial update by using a bool. And add using HoloToolkit.Unity.SpatialMapping; to get access to the Spatial mapping classes.
  • Now, you’ll have your own spatial data.

And you can use the same method to place your hologram in the spatial data you’ve created depending on your needs.

All you have to do is just scan the room and air tap to stop the scan. There you go, a scanned room with exact details of the room.

1 Like