Minimal Example of Updating Sky Reflection and Ambient SH

Is there a minimal example, anywhere, that shows how to update a procedural skybox such that objects in the scene are updating their sky environment reflection and ambient (SH) light per-frame as the sunlight rotates.

Consider a scene containing only a camera, a directional light, and a sphere. Suppose that the only directional light (the sun) is set to an intensity of 0.01 so it is almost negligible, and we make the atmosphere on the skybox really thick to exaggerate it

. Suppose that in edit mode the scene looks like [screenshot 1] this when the sun is low to the horizon, and [screenshot 2] when it’s overhead, and assume that all lighting in the scene is coming from environment lighting and environment reflections (which is easy to prove by turning their lighting settings to zero.) We’re not in PlayMode, “auto generate lighting is checked”, and the only difference between the screenshots is that light.Transform.x is changed from 0.0 to 90.0.

Now, what exactly do we have to do to make that lighting update identically at runtime, in playmode, on an arbitrary device (possibly ios, etc.) and have the environment lighting (presumbaly internally an SH), and environment reflections update identically to edit mode? Googling reveals a massive number of hits for people trying LightProbes, ReflectionProbes, changing bits of qualitysettings, calling ReflectionProbe.RenderProbes(), etc.

We cannot replicate the behavior identically between the simple case in the editor when auto-update is enabled, and runtime with any combination of things. Half the posts read like people trying things at random then suggesting things work

Things that are clearly not suffient:
Enabling DynamicGI.SynchronousMode.
Calling DynamicGI.UpdateEnvironment

Enabling Dynamic Reflection Probes in PlayerSettings and creating a reflection probe which we set to “realtime”.

It feels like a simple example of placing a sphere in the scene and updating environment as the sun moves should be available in a minimally correct form somewhere.

Unity 2018.4.x.

Nobody?

I still can’t get a minimal example of a reflective metallic sphere to update the sky reflection texture at runtime. The diffuse case seems to work OK provided you call DynamicGI.UpdateEnvironment().

What am I missing?

I guess you are missing the specular part, ie from an up to date reflection probe :slight_smile:

Thanks for the reply. I have made a working version of this since posting, but I’m not happy about the fact that I can’t explain why it’s correct, or enough about the interaction of the standard shader with sky/reflection (note we’re on mobile so not actually using GI) to satisfy myself it works in all cases. And I’d like to leave this thread in a state such that future graphics programmers who stumble across it can derive all the details (I’m less concerned about lighting artists, and more with programmers trying to understand Unity’s lighting and which terms of it are affected by what algorithmically.)

Yes, I have now made a version that seems to work correctly, but I still feel the docs, or at least the ones I’ve seen, don’t do a great job of explaining how the algorithmic-level terms of the lighting are handled or updated. The docs for, e.g. DynamicGI.UpdateEnvironment() say “Schedules an update of the environment texture”, which is super-vague. Which textures? Does is also recompute ambient SH? When does it schedule it for?

Here’s a screenshot of nine spheres (with values of 0, 0.5, 1.0 for metallic and specular.) I’ve verified that I can make them look the same under various rotations of a single light (which the procedural skybox uses as its sun) both in editor with “Auto Update” enabled and in play mode. I’m assuming device will behave identically to play mode with auto-update disabled, but it’s harder to diff screenshots then…

Through experimentation, I conclude that:

(1) We must create a Reflection Probe. The sky itself does not update as a reflection probe (?? - what does that mean - update a convolved specular texture, or ???) when you move the light. Note that I can see the sun moving in the metallic objects even if I don’t use a reflection probe, so I assume that there are multiple levels of texture here - ? the skybox itself which always gets updated and a convolved version created for the sky by baking but only updated at runtime when using a reflection probe?
(2) Calling DynamicGI.UpdateEnvironment is also necessary after moving the light even when using a reflection probe otherwise some term isn’t updated ??ambient SH?? (see screenshot 2, composed by moving the light after the build has started, and clearly some term of the lighting is different.)

I don’t really understand (a) why there isn’t an algorithmic-level description of which function calls update the SH component, which things update convolved map foo and which things update convolved map bar, and which maps the standard shader samples.

I don’t really understand why the sky doesn’t behave like a reflection probe, given that we have moved the light and called DynamicGI.UpdateEnvironment().

As a similar example, in this case I’m assuming DynamicGI.UpdateEnvironment update the SH coefficients used for irradiance, but I’m not using a LightProbe Group to do this, so why do I need a reflection probe to duplicate the sky functionality but not a light probe?

I don’t really understand why the only way to derive all of this stuff is to read the standard shader source code, call functions, and see which maps and coefficients change when you fiddle with stuff? Is there a page of the docs that I’m missing? The more I read shaders, everything for the sky/envmap seems to be passed in the 9-component SH evaluated as ambient, unity_SpecCube0 (is this ultimately a single envmap, and if so why is the sky/refl probe independent as it seems to be), and unity_SpecCube1 (used only with probe interpolation, and not used in my case.) Assuming I’m correct and in the general (mobile) case all lighting is reduced to ambient SH and unity_SpecCube0, there seems to be almost no docs about how/what objects in the scene updates those values at runtime.

3 Likes

This must be in the docs for sure
It would be great to have some API to control this internal reflection probe, at least trigger an async bake as it is done automatically in the editor

2 Likes

We’re seeing this problem in our game (on 2019.3.0f3, currently - PC platform/HDRP). We see it with both Procedural Sky and Physical Sky.
The specular reflection from sky appears fine and up-to-date at all times, but diffuse catches up only at unpredictable times, triggered by unknown events, or never.
I’ve thrown everything I could find at it, but had no luck. It sure feels like a bug.
Would be nice to have some clarity on this, as it’s a significant problem for us.
Edit: standalone build does not seem to exhibit the issue, so maybe this problem is in editor only.

Hi

Could you maybe make a small repo project and file this as a bug and we can have a look? Thanks :slight_smile:

i may have the same or a similar issue. please tell me if i misunderstood the current test case discussed here.

my situation:
simple space game, very empty scene, on load of scene i change 1 skybox face.
(important hint: as it is a shared material, leaving the PLAY mode and returning to editor it stays the way i changed it until i enter PLAY mode next time).

after changing 1 skybox face, calling DynamicGI.UpdateEnvironment seem NOT to update the (diffuse spherical harmonics) light probes.

(i am on unity 2019.3.4f1, building for android, testing with play mode on windows, this behaviour is consistent between play mode and android buil)

what i am doing in detail:
-empty space scene (black skybox textures )
-one face of the skybox material gets exchanged against colorful texture (purple, green, etc.)
-the exchange happens in a script on MonoBehaviour.Start()
-when the game is running (MS Windows or Android build) the spherical harmonics still show what was PREVIOUSLY visible (in the editor / before the build)
-it is NOT updating the spherical harmonics, neither when calling DynamicGI.UpdateEnvironment () nor at any time later (waited for minutes).
-i also changed all other DynamicGI settings according to documentation, to ensure that it definitely gets updated
-the scene is very empty, it shouldnt be a performance issue
-nothing happens while running

excerpt from my code

RenderSettings.skybox.SetTexture(active_face, stars[Random.Range(1, stars.Length)]);

DynamicGI.materialUpdateTimeSlice = 5000; //5 seconds
DynamicGI.synchronousMode = true;
DynamicGI.UpdateEnvironment();

while(DynamicGI.isConverged == false)
    Debug.Log("waiting for GI to converge");

the Debug.Log message is never shown, so i believe it is not a performance / waiting problem.

thus it seems the documentation is wrong and the call to DynamicGI.UpdateEnvironment does not update the ambient probes?

addendum:

  • all my objects are only lit by 1 direction light in scene + 1 very range limited point light + 1 light probe group with 32 light probe spheres inside (spread evenly as a grid of 4x4x2 around <0,0,0>)

  • i have a custom shader that works well but allows to make the SH term to be even more clearly visible.

-i have static and dynamic meshes

-i have zero lightmapped objects

-if i light map one of my static meshes, the load time of the scene goes definitely up by a second or two, and the object propery shows the current skybox face GI (e.g. pink or green light is visible on correct side of otherwhise dark-shadowwed part of the static mesh)

-even if i add such a lightmap, the spherical harmonics still seem unlit / unchanged / unupdated

and more information:

  • i am using progressive cpu lightmapper
  • the scene has no fog
  • mixed lighting has baked GI with shadowmask
  • i have checked that the probes are properly used by my meshes, using the Lighting > Scene > Debug Settings > Light Probe Visualization

and more:

  • i am NOT using Lighting > Scene > Realtime Lighting > Realtime GI (Deprecated)
  • i am using Mixed Lighting > Baked Global Illumination

… is this perhaps the problem? it doesnt make any sense to turn it of to only have realtime lights and not mixed anymore (which i believe i want and need), but it seems to solve at least the issue that the probes are not getting updated anymore. is that possible?

and:

it feels totall silly and wrong:

  • if i turn OFF Lighting > Scene > Mixed ighting > Baked GI then the spherical harmonics are updated as expected (but all lightmapping is gone, is that correct? or my interpretation?)
  • if i turn it OFF also the preview of spherical harmonics: Light > Scene > Debug > Light Probe Visualization doesnt show any probes anymore connected to my test mesh, in fact for no mesh at all. as if the feature of light probes via SHs is now disabled (which i believe visually i can tell it is not)
  • if i turn it ON the preview of SH is showing me reasonable ones and the light looks good (while in editor) and in PLAY mode it doesnt update the probes anymore. in fact once i return to editor i can see for a few frames how it looked while in PLAY mode (just from different editor camera perspective) and then the lighting updates to how the scene was randomized (in PLAY mode)

i have the feeling that

  1. documentation is lacking a lot here to clarify things
  2. editor visualization can have a bug
  3. baked GI setting in editor can have a bug
  4. DynamicGI.Update either has a bug or needs more debug information so its clear what is happening

please help me to solve what i am doing wrong (or misunderstood) and verify what seem to be bugs in unity

Hi, I have brought this up with the documentation team. I agree that the current state of the ambient docs leave a lot to be desired and we will improve this so your questions can be answered.

Thank you for checking the forum and following up.

I’ve run into this issue too. A solution I’ve come up with is to

  • Set the RenderSettings Reflection to Custom
  • Create a new Cubemap() via script, assign it to RenderSettings.customReflection
  • Create a RenderTexture matching the Cubemap format
  • Use a command buffer to render each face into the RenderTexture using the skybox shader, using a Sphere
  • Copy each Face into the final Cubemap

A basic example is below. A few notes:

  • It uses the built-in sphere mesh. Unity’s default skybox renders with a smoother mesh which gives better results
  • Mipmaps only use bilinear-downsampling which does not match built-in reflection probes, as they use a special shader.
  • This will generate an uncompressed texture, unlike the built-in probe. However it’s possible to use BC6 format and a ComputeShader to compress the Cubemap, which will help with rendering.
  • Realtime reflection probes render and convolve the cubemap faces over several frames which my example doesn’t do. You will want to do this for performance reasons.

A couple of advantages however:

  • You have control over some built-in parameters such as camera position, light color/direction etc.

  • You can enable/disable keywords, for example, you might not want to render the sun, as it will be rendered as a light source during regular rendering.

  • I haven’t yet tested, but I’m hoping this will also stop it being included in ambient lighting after calling DynamicGI.UpdateEnvironment too

  • As mentioned above, you can compress the texture at runtime for faster rendering.

using UnityEngine;
using UnityEngine.Rendering;

[ExecuteAlways]
public class EnvironmentUpdater : MonoBehaviour
{
    private CommandBuffer commandBuffer;
    private Cubemap cubemap;
    private RenderTexture renderTexture;

    private void OnEnable()
    {
        // Initialize cubemap/render texture
        RenderSettings.defaultReflectionMode = DefaultReflectionMode.Custom;
        var resolution = RenderSettings.defaultReflectionResolution;
        renderTexture = new RenderTexture(resolution, resolution, 0, RenderTextureFormat.ARGBHalf) { autoGenerateMips = false, useMipMap = true };
        renderTexture.Create();

        var skyboxMaterial = RenderSettings.skybox;
        cubemap = new Cubemap(resolution, TextureFormat.RGBAHalf, true) { filterMode = FilterMode.Trilinear };
        cubemap.Apply(false, true);

        RenderSettings.customReflection = cubemap;

        // Initialize command buffer
        var mesh = Resources.GetBuiltinResource<Mesh>("Sphere.fbx");
        commandBuffer = new CommandBuffer();

        var projectionMatrix = GL.GetGPUProjectionMatrix(Matrix4x4.Perspective(90, 1, 0.1f, 1), true);
        commandBuffer.SetProjectionMatrix(projectionMatrix);

        // Matrices for rendering the six cubemap faces
        var matrices = new[] { Matrix4x4.TRS(Vector3.zero, Quaternion.LookRotation(Vector3.right, Vector3.down), -Vector3.one).inverse,
        Matrix4x4.TRS(Vector3.zero, Quaternion.LookRotation(Vector3.left, Vector3.down), -Vector3.one).inverse,
        Matrix4x4.TRS(Vector3.zero, Quaternion.LookRotation(Vector3.up, Vector3.forward), -Vector3.one).inverse,
        Matrix4x4.TRS(Vector3.zero, Quaternion.LookRotation(Vector3.down, Vector3.back), -Vector3.one).inverse,
        Matrix4x4.TRS(Vector3.zero, Quaternion.LookRotation(Vector3.forward, Vector3.down), -Vector3.one).inverse,
        Matrix4x4.TRS(Vector3.zero, Quaternion.LookRotation(Vector3.back, Vector3.down), -Vector3.one).inverse };

        // Set the camera to render each face into a temporary texture, and then copy that texture into the final cubemap
        for (var face = CubemapFace.PositiveX; (int)face < 6; face++)
        {
            commandBuffer.SetViewMatrix(matrices[(int)face]);
            commandBuffer.SetRenderTarget(renderTexture, 0);
            commandBuffer.ClearRenderTarget(true, true, Color.clear);
            commandBuffer.DrawMesh(mesh, Matrix4x4.identity, skyboxMaterial);
            commandBuffer.GenerateMips(renderTexture);
            commandBuffer.CopyTexture(renderTexture, 0,cubemap, (int)face);
        }
    }

    private void Update()
    {
        Graphics.ExecuteCommandBuffer(commandBuffer);
    }
}
1 Like

I just remembered you can access the global ReflectionProbe with ReflectionProbe.defaultTexture, assuming you have it already baked in-Editor. So you could use this to avoid having to manually create a cubemap. Keep in mind that it may be compressed depending on your settings, so you’ll have to set it to uncompressed first.

Did any documentation ever get updated here? We’re testing an XR app that instantiates a lot of content, with variable orientations (depending on where the user was looking) and a lot of very finely tuned lighting, and it would be great to say to our client “yes, we’re 100% sure everything is right on device.”

The ticket’s still open with the documentation team. Sorry for the delay.

Hey Kasper, is there any updated info on this? I believe I’ve reverse-engineered everything I need to know at this point, but it would be great if there were an official explanation.

Thanks,
Alex

Hi, the ticket is still with the docs team. It is not forgotten, they are just swamped with requests. I am sorry that it is taking so long.