Realtime Lightmap - What is that exactly?

[ZosimusJulius]

Hi there

I hope anyone could bring some light onto this subject, as I am not quite sure if I do fully understand what’s going on behind the scenes and the documentation on this subject is rather vague.

I’m currently experimenting with lightmapped prefabs (something like: Problems with instantiating baked prefabs ). Basically, I bake the prefabs in a scene and serialize on it the reference to the baked lightmap. On instantiation, the lightmap then gets added to the LightmapSettings.lightmaps array and the renderer of the prefab will point to that map by the setting the lightmapIndex property. This works fine, no matter in which scene the prefab will get instantiated.
Now, I’d like to use realtime GI in my project and therefore will bake the lightmaps for the prefab with the Realtime Global Illumination checkbox enabled. As expected, the ambient light does not get baked onto the lightmap anymore. But if I go ahead instantiating the prefab in a scene, which is not the one it was baked in, the lightmap seems to get wrongly interpreted as it renders the parts that should be affected by the realtime GI / ambient light completely black (as it is, in the baked texture). So there seems some information is going missing.

I noticed the renderer component also contains the properties realtimeLightmapIndex and realtimeLightmapScaleOffset. From the name I would assume, those refer to some realtime lighting data. However, the documentation does not really help there. I have no clue, what a realtime lightmap exactly is. Well, a lightmap that gets rendered realtime while running, I guess.
Clearly, the realtimeLightmapIndex is not used as index for the LightmapSettings.lightmaps array.

So, how exactly does a ‘realtime lightmap’ work? Where is this index used and how do I know, to which value it is to be set (e.g. in a different scene, not the one it was baked in)? Am I on the correct path for making realtime GI working, or am I completely wrong and the data I am looking for is somewhere different, maybe even inaccessible?

Would really appreciate some comments on this
Cheers

[JenniferNordwall]

Hi

Had to do some research into the code, since this is usually not something people do.

So when it comes to realtime GI, all this data lives as chunks in the Lighting Data Asset, and lightmaps are then generated on the fly (so that it can account to changes in materials and light setup). When we patch up the renderers for realtime GI, we set these 3 things:
realtimeLightmapIndex
realtimeLightmapScaleOffset
AdditionalVertexStreams

So my suggestion would be to print out all these properties after a successful bake, save it somewhere and reapply it. I haven’t tested it myself but please let me know if it works for you.

As for the scene bit, I do not think this will work, since the Lighting Data Asset data is scene specific. But what you can do is to load a scene and it’s specific realtime data, add in a few objects and force them to use indexes and STs that makes sense to the scene data. Might be had to get it right, and not sure what exactly your use case is?

Cheers
Jennifer Nordwall
Lighting Team

[ZosimusJulius]

Thank you so much for this clarification, that helped a lot!

@JenniferNordwall

So when it comes to realtime GI, all this data lives as chunks in the Lighting Data Asset, and lightmaps are then generated on the fly

That’s what I expected. Even found the corresponding data after looking through the hex-dump of the asset. Makes sense imo.

@JenniferNordwall

When we patch up the renderers for realtime GI, we set these 3 things:
realtimeLightmapIndex
realtimeLightmapScaleOffset
AdditionalVertexStreams

additionalVertexStreams, that’s what I missed! Storing this data and re-applying works very well now. Am I correct in the assumption that these values do not change on runtime through an updating realtime lightmap or something? Meaning, I can safely store all this data on a prefab after baking, and re-applying them on instantiation. → This is working fine in my tests, but just to make sure…

And yes, as for the scene bit, I will have to improvise. I think I am going simple with the lightmapped scene as ‘root’ scene and then load in others additively, keeping newly instantiated objects on the scene they’re baked on.

Thanks again for the support!

[JenniferNordwall]

Hi

From what I can see, we are not changing these things after the bake is done. Happy to be of help

[skycc]

@AlbinZosimusJulius

Thank you so much for this clarification, that helped a lot!

That’s what I expected. Even found the corresponding data after looking through the hex-dump of the asset. Makes sense imo.

additionalVertexStreams, that’s what I missed! Storing this data and re-applying works very well now. Am I correct in the assumption that these values do not change on runtime through an updating realtime lightmap or something? Meaning, I can safely store all this data on a prefab after baking, and re-applying them on instantiation. → This is working fine in my tests, but just to make sure…

And yes, as for the scene bit, I will have to improvise. I think I am going simple with the lightmapped scene as ‘root’ scene and then load in others additively, keeping newly instantiated objects on the scene they’re baked on.

Thanks again for the support!

hi AlbinZosimusJulius,did you successfully restore realtime lightmap?
which version of unity do you use?
i’m doing the same thing,but the realtimeLightmapIndex of any renderer always return 0 even if there is many realtimelightmap textures,i just saved realtimeLightmapIndex,realtimeLightmapScaleOffset,the AdditionalVertexStreams,and re-apply them,but it didn’t work,looks like the renderer is not linked with the realtimelightmap
i’m using unity2018.3.0f2

[ZosimusJulius]

Hey skycc,

Your approach did certainly work for me and I believe it also should work for you too, so I’ll try to elaborate on what I exactly did in my case. I’m using Unity 2018.1.x by the way. There were some small changes in the Scripting API regarding lightmapping but that should not interfere, I believe.

So I did the following:
I triggered the lightmap bake process obviously from an editor script. Then, right after the bake has completed, checking the mesh renderer(s) of the object (prefab) and save all of the data: lightmapIndex, lightmapScaleOffset, realtimeLightmapIndex, realtimeLightmapScaleOffset and additionalVertexStreams. Now, pay attention to the last one → the vertex streams are stored as a mesh instance. Meaning this is saved by reference to an internal mesh object, which might not be available after you instantiate the prefab on runtime. You can verify that by checking the reference, which will likely be null. In order to safely store the mesh data you therefore need to create a deep copy of the Mesh object that should be accessible to the prefab. The simplest way of doing so is with something like:

// ...

const string assetPath = "some/path/for/saving/meshdata.asset";
Mesh meshData;

// 'renderer' is the mesh renderer component
AssetDatabase.CreateAsset(renderer.additionalVertexStreams, assetPath);
AssetDatabase.ImportAsset(assetPath); // not sure if needed...
meshData = AssetDatabase.LoadAssetAtPath<Mesh>(assetPath);

// Now meshData points to an asset and can be referenced by a prefab

// ...

Re-apply all the data to the prefab in Awake → that should work.

Don’t worry about the realtimeLightmapIndex being 0. This is a valid index and multiple objects can certainly point to the same lighmap. You basically have no control over how Unity bakes and applies those lightmaps and you can not copy the realtime lightmap directly, so there’s one per-scene limitation: You need to bake the realtime GI data in one go for all prefabs per scene. On every bake, the previous realtime lightmap data is destroyed (lies in LightmapData asset), meaning the previously saved data for your prefab will now be obsolete.

Because of this I wrote an editor-tool for our project that sets up the prefabs for the target scene and then starts the baking process and also serializes the baked data onto the prefab. If lighting changes on one object, all objects for this scene have to be rebaked. Of course some optimizations in the artists workflow can speed up this process.

I hope I’ve been able to shed some light on the matter.
Cheers

[jister]

Hey
Doing the same thing in 2019.3 and for some reason the mesh is ok at first but becomes missing when i place the Prefab in the scene.
Also they still show the mesh icon and are assets on disk, but in the project folder they seem to be broken some how because once they go missing I can’t delete them anymore?

1. Cache right after baking
   
   
2. After making it a variant prefab
   
   
3. in a new scene
   
   

Script:

using UnityEngine;
#if UNITY_EDITOR
using UnityEditor;
#endif

//[RequireComponent(typeof(MeshRenderer))]
public class RealTimeGICache : MonoBehaviour
{
    public bool autoReapply;
    public int[] RTLightmapIndex;
    public Vector4[] RTLightmapOffset;
    public Mesh[] RTLightmapVertex;
    public MeshRenderer[] render;
    private int counter;

    private void Awake()
    {
        render = GetComponentsInChildren<MeshRenderer>();
        if(autoReapply && RTLightmapVertex != null && RTLightmapOffset != null && RTLightmapIndex != null)
            ReapplyRealTimeLightData();
    }
#if UNITY_EDITOR
    public void CacheRealTimeLightData(string assetPath)
    {
        if (string.IsNullOrEmpty(assetPath))
            assetPath = "Assets/GIMeshes";
        if(!AssetDatabase.IsValidFolder(assetPath))
            AssetDatabase.CreateFolder("Assets","GIMeshes");
        render = GetComponentsInChildren<MeshRenderer>();
        RTLightmapIndex = new int[render.Length];
        RTLightmapOffset = new Vector4[render.Length];
        RTLightmapVertex = new Mesh[render.Length];
        for (int i = 0; i < render.Length; i++)
        {
            RTLightmapIndex[i] = render[i].realtimeLightmapIndex;
            RTLightmapOffset[i] = render[i].realtimeLightmapScaleOffset;
            string fullPath = assetPath + "/RTGIMesh" + (counter++) + ".asset";
            Debugger.Log("Path: "+fullPath);
            render[i].additionalVertexStreams.hideFlags = HideFlags.None;
            AssetDatabase.CreateAsset(render[i].additionalVertexStreams, fullPath);
            AssetDatabase.SaveAssets();
            AssetDatabase.Refresh();
            //UnityEditor.AssetDatabase.ImportAsset(assetPath);
            RTLightmapVertex[i] = AssetDatabase.LoadAssetAtPath<Mesh>(fullPath);
        }
    }
    #endif
    public void ReapplyRealTimeLightData()
    {
        render = GetComponentsInChildren<MeshRenderer>();
        for (int i = 0; i < render.Length; i++)
        {
            render[i].realtimeLightmapIndex = RTLightmapIndex[i];
            render[i].realtimeLightmapScaleOffset = RTLightmapOffset[i];
            render[i].additionalVertexStreams = RTLightmapVertex[i];
        }
    }
}

Editor script

using UnityEngine;
using UnityEditor;

[CustomEditor(typeof(RealTimeGICache)), CanEditMultipleObjects]
public class RealTimeGICacheEditor : Editor
{
    public static string assetPath = "Assets/GIMeshes";
    public override void OnInspectorGUI()
    {
        serializedObject.Update();
        assetPath = EditorGUILayout.TextField(assetPath);
        RealTimeGICache RTGI = (RealTimeGICache)target;
        if (GUILayout.Button("Cache Real Time Data"))
            RTGI.CacheRealTimeLightData(assetPath);
        if (GUILayout.Button("Reapply Real Time Data"))
            RTGI.ReapplyRealTimeLightData();
        serializedObject.ApplyModifiedProperties();

        base.OnInspectorGUI();
    }
}

[jister]

Also render[i].additionalVertexStreams.vertices.Length prints 0
while render[i].additionalVertexStreams.vertexCount prints some number
aren’t they suppose to be the same?
@JenniferNordwall

[jister]

@JenniferNordwall

As for the scene bit, I do not think this will work, since the Lighting Data Asset data is scene specific.

Ah i overlooked this part. I was trying to reapply the GI data in an other scene then the original was baked. That doesn’t work, probably because the data I cached wasn’t found in an other scene’s lighting data…?

[jister]

Still have the same problem: the saved mesh assets become broken/missing after going in and out of play mode using the same scene as they where baked in…?
@AlbinZosimusJulius Did you experience similar problems?

[ZosimusJulius]

Hi jister
Sorry for the late answer, I didn’t realize the mention. I’m not really aware of how things work with newer versions of Unity, and I haven’t really looked into lightmapping again since. My wild guess for your problem would be, that it has a per-scene reference from your prefab (or variant) to the mesh asset → you never apply your prefab. In the code you provided, you save a copy of the mesh data as file and reference that. In the end of the day you save the whole thing by calling serializedObject.ApplyModifiedProperties();. That’s a reference saved in your scene-file, and gets lost if you remove your prefab. Apply the prefab or variant and see if that makes a difference. No guarantee though
Cheers

[SinDeSiecle]

I have an interesting experience with this. In my main game scene objects are instantiated at runtime on a spherical world. Instead of storing data on prefab, I tried to load scene additive by addressables. Now, the sequence is like this: the object that loads the addressable scene is instantiated at runtime. This is just an empty game object with a script that loads the addressable scene on it. This loads the targeted scene additively, re-positions the root object in that scene to its position and also matches the additively loaded scene object’s rotation to its rotation. Then, LightProbes.TerraHeralize() but this is irrelevant. Realtime lightmaps are working, but here is the problem:

1- If I reposition it at runtime like this, realtime lightmaps only works within the initial spawned area (in the area before re-positioning). Also light probes seem to be working there.

2- If I rebake the prefab in the additively loaded scene, but this time, re-positioning it here with the delta values gathered from the main scene reposition operation, so that when the scene is loaded additively, no repositioning will be needed, everything works fine in every single part of the prefab.

This tells me that on additive scene loading, the initial spawn area is set for the realtime lightmaps and any rotation or translation will take the prefab off that area. Also, when this happens, if I walk outside that building prefab with my in-game character, the illumination of the interiors are applied to the character on outside environment as long as I stay in that initial spawn area of the prefab. Therefore, accurate positioning in the prefab scene spawns it as it should be in the main game scene, and everything works well.

Now, I am looking for a solution to apply realtime GI data after re-positioning of the gameObject in the additively loaded scene. So far, storing realtime GI data and re-applying it did not work.

One interesting experiment on this: I tested this re-positioning process on the prefab scene only. Run the scene → go to the scene view → translate the object by 1000 units. Everything works fine. Now do a second test and instead of translating the object, rotate it 10 - 15 degrees around x or z so that it will have some tilt. The parts of the building outside the initial rotation begin to loose the realtime GI data and character is not illuminated anymore in those parts. I don’t know if it is it a bug with GI data, but, it seems translation does not cause the same thing.

Attaches photos show this phenomenon in my game scene. Screenshot (5) shows the initial spawn rotation (the cubes) and the re-positioned rotation (the building). In Screenshot (3), it is possible to see the outer surface of the cube, so we are fully sure that we are out side that initial spawn area and our character is not illuminated. Screenshot (4) is taken inside that cube, the character is illuminated.

So, I am looking for a solution to this phenomenon. I already have a half-solution as it is possible to re-position the object in its own scene relative to its actual position in the game scene. One more trick on this in this my game setup should do the job, but I’d prefer a full solution rather than that trick. At least, confirmation of this behavior being an unintended behavior would lead me to develop short term - mid term - long term strategies on this issue so that I can continue to work on the rest of the game and define a safe release date.

Any suggestions?

[jister]

I solved my problem with adding UploadMeshData(true);

render[i].additionalVertexStreams.hideFlags = HideFlags.None;
            Mesh m = new Mesh();
            m = render[i].additionalVertexStreams;
            m.UploadMeshData(true);
            AssetDatabase.CreateAsset(m, fullPath);
            AssetDatabase.SaveAssets();
            AssetDatabase.Refresh();
            UnityEditor.AssetDatabase.ImportAsset(assetPath);

[jister]

@jister

I solved my problem with adding UploadMeshData(true);

render[i].additionalVertexStreams.hideFlags = HideFlags.None;
            Mesh m = new Mesh();
            m = render[i].additionalVertexStreams;
            m.UploadMeshData(true);
            AssetDatabase.CreateAsset(m, fullPath);
            AssetDatabase.SaveAssets();
            AssetDatabase.Refresh();
            UnityEditor.AssetDatabase.ImportAsset(assetPath);

ok that didn’t solve my problem!
I got some strange things going on.
When i just save the GI vertex stream to an asset, it breaks after going in and out of play mode. (see above)
When i duplicate it manually (crtl D) then the Asset stays ok after in/out playmode!
If i duplicate it by code then it breaks again after in/out playmode!?

Any idea whats going on here?
I really need to get this solved so any suggestions welcome!