ARFoundation & ARCore HDR light estimation

coder89 · October 6, 2021, 12:25am

Hi,

I’m working on a project where I try to enable HDR light estimation. On a test device I got (some high-end android phone) I run arfoundation-samples app and I realized that mainLightColor and mainLightIntensity properties are reporting strange values (or at least values I did not expect and/or lacking proper documentation). So…

mainLightColor - in bright rooms RGB values often get values outside of 0-1 values (i.e. it is not that uncommon to get values above 2.0-3.0 float for each channel). Alpha channel is always set to 1.0.
mainLightIntensity - it is always undefined.
ambientProbe - can’t really tell whether values are correct or not…

So my questions are:

How should I handle mainLightColor properly? I mean how should I set that on my directional light’s color property given it’s values are outside of expected range (that is outside 0-1 range)? If I pass color value “as-is” objects on the scene render way too bright so that all bright textures turn completely white and black turns grey.
Why there is no mainLightIntensity?

What am I missing here (and what is missing in the official arfoundation-sample) that would help me to get things working correctly?

Oh and I am running build-in render pipeline if that makes any difference.

Thanks in advance!
/Lukasz

coder89 · October 6, 2021, 7:29am

Slept just a couple of hours 'cause I couldn’t stop thinking about this annoying issue

So… this is what I found in ARCore’s SDK repo:

github.com

google-ar/arcore-unity-sdk/blob/2ad81756cd9b147bc9b21e4a09f9b348f99a5b12/Assets/GoogleARCore/SDK/Scripts/Api/Wrappers/LightEstimateApi.cs#L113


      
              lightColor = Color.black;
          
              ExternApi.ArLightEstimate_getEnvironmentalHdrMainLightIntensity(sessionHandle,
                  lightEstimateHandle, _tempColor);
              lightColor.r = _tempColor[0];
              lightColor.g = _tempColor[1];
              lightColor.b = _tempColor[2];
          
              // Apply the energy conservation term to the light color directly since Unity doesn't
              // have that term in their PBR shader.
              lightColor = lightColor / Mathf.PI;
          
              ExternApi.ArLightEstimate_getEnvironmentalHdrMainLightDirection(sessionHandle,
                  lightEstimateHandle, _tempVector);
              Vector3 lightDirection = Vector3.one;
              ConversionHelper.ApiVectorToUnityVector(_tempVector, out lightDirection);
          
              // The ARCore output the light direction defined for shader usage: lightPos-pixelPos
              // We need to invert the direction to set it Unity world space.
              lightRotation = Quaternion.LookRotation(-lightDirection);
          }

And here is another one

github.com

google-ar/sceneform-android-sdk/blob/b33fa25987561e5bf2129a25fbce6ad3b43ac4ee/sceneformsrc/sceneform/src/main/java/com/google/ar/sceneform/ArSceneView.java#L633


      
            lastValidEnvironmentalHdrMainLightIntensity = mainLightIntensity;
          }
          
          if (lastValidEnvironmentalHdrAnchor == null
              || lastValidEnvironmentalHdrMainLightIntensity == null
              || lastValidEnvironmentalHdrAmbientSphericalHarmonics == null
              || lastValidEnvironmentalHdrMainLightDirection == null) {
            return;
          }
          
          float mainLightIntensityScalar =
              Math.max(
                  1.0f,
                  Math.max(
                      Math.max(
                          lastValidEnvironmentalHdrMainLightIntensity[0],
                          lastValidEnvironmentalHdrMainLightIntensity[1]),
                      lastValidEnvironmentalHdrMainLightIntensity[2]));
          
          final Color mainLightColor =
              new Color(

Could someone from Unity team could confirm whether something needs to be done for ARCore devices when HDR light estimation is used along with Build-in render pipeline & standard shaders?

Also, do I need to use tonemapping from the post-processing stack v2 in order to get the colors right or simple color normalization (link #2) is good enough?

coder89 · October 6, 2021, 8:41am

And finally I think I found a more complete explanation of how HDR lightning is supposed to be used in ARCore (not sure if it’s the same on ARKit though):

github.com

google-ar/arcore-android-sdk/blob/3847878dae324c53f5dace1a4b1736df9ce070c9/samples/ml_kotlin/app/src/main/assets/shaders/environmental_hdr.frag

#version 300 es
/*
 * Copyright 2020 Google LLC
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
precision mediump float;

// This shader will light scenes based on ARCore's Environmental HDR mode with a
// physically based rendering model.

This file has been truncated. show original

ALso not completely sure though how to implement that in Unity but my guess it that scale light by “Mathf.Pi” and then normalize would be the way to go. Could someone confirm whether my thinking is correct or not?

Looks like this fix applied to the samples is no longer valid?
https://github.com/Unity-Technologies/arfoundation-demos/commit/a62edf595b039420501cfc1d57b3430c147df530

Treyrannosaurus-Rex · October 18, 2021, 7:26pm

Team had a look and had this guidance to share:

The energy conservation is already applied within ARFoudnation APIs. Above 1 values are not uncommon in HDR.

Topic		Replies	Views
ARFoundation - Enabling Light Estimation Unity Engine XR , AR-Foundation	26	20264	March 27, 2021
Cannot read 'light estimation' values Unity Engine XR , AR-Foundation , Question	5	1461	November 2, 2021
How do i get the correct light estimation in AR Foundation? Questions & Answers legacy-topics	1	1645	February 27, 2020
ARFoundations (ARCore) missing values in light estimation Unity Engine XR , AR-Foundation , Question	1	912	January 25, 2022
ARCore User face tracking and mainLightDirection estimation Unity Engine XR , AR-Foundation , Question	3	1721	January 5, 2021

ARFoundation & ARCore HDR light estimation

Related topics