Support for UI elements in RenderToCubemap

I know this is not presently possible, but I would like to see an option in the RenderToCubemap() function that would allow you to capture UI elements. I use this function for non reflection probe based work, and process cubemap data to create 360 equirectangular images.

I dont want to open a BUG ticket in Unity, so I thought maybe someone on the graphics team at Unity could see this.

Ideally, I would want to have the UI captured in the world forward camera direction for worldspace items.

Thanks

Could also submit a feedback request here
https://feedback.unity3d.com/

As the bug is still not tackled, I added a feedback request: https://feedback.unity3d.com/suggestions/support-for-world-space-ui-elements-in-rendertocubemap Feel free to vote for it! :slight_smile:

I ran into the same problem trying to render a world-space UI to a cubemap in 2017.2. I ended up throwing together a script to manually draw all UI elements in the scene. I couldn’t find any documentation on how the UI system does its rendering, so don’t expect this to work if you’re using non-standard UI components. It’s also pretty slow and uses some hacks that might not work on other versions.

using System.Linq;
using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.UI;

/// <summary>
/// Component that manually renders world-space UI Graphics to a Camera. Intended for
/// use with Camera.RenderToCubemap which seems to ignore the built-in UI system
/// </summary>
[RequireComponent(typeof(Camera))]
public class ManualUIRenderer : MonoBehaviour
{
    private static readonly int MainTexProperty = Shader.PropertyToID("_MainTex");
    private static readonly int TextureSampleAddProperty = Shader.PropertyToID("_TextureSampleAdd");
    private static readonly int ColorProperty = Shader.PropertyToID("_Color");

    private Camera targetCamera;
    private CommandBuffer commandBuffer;

    private void Awake()
    {
        this.targetCamera = this.GetComponent<Camera>();

        this.commandBuffer = new CommandBuffer();
        this.commandBuffer.name = "Manual UI rendering";
        this.targetCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);
    }

    private void OnDisable()
    {
        this.commandBuffer.Clear();
    }

    private void Update()
    {
        this.commandBuffer.Clear();
        AddUiDrawingCommands(this.targetCamera, this.commandBuffer);
    }

    private static void AddUiDrawingCommands(Camera cam, CommandBuffer buffer)
    {
        // Root canvases ordered by screen space depth
        var rootCanvases = FindObjectsOfType<Canvas>()
            .Where(canvas => canvas.isRootCanvas && canvas.renderMode == RenderMode.WorldSpace)
            .OrderByDescending(canvas => cam.WorldToScreenPoint(canvas.transform.position).z);

        foreach (var canvas in rootCanvases)
        {
            // Graphics after culling sorted by depth
            var graphics = canvas.GetComponentsInChildren<Graphic>()
                .Where(graphic => TestCullingMask(graphic, cam.cullingMask))
                .OrderBy(graphic => graphic.depth);

            foreach (Graphic graphic in graphics)
            {
                AddGraphicDrawingCommands(graphic, buffer);
            }
        }
    }

    private static bool TestCullingMask(Graphic graphic, int cullingMask)
    {
        return ((1 << graphic.gameObject.layer) & cullingMask) != 0;
    }

    private static void AddGraphicDrawingCommands(Graphic graphic, CommandBuffer buffer)
    {
        // Probably not needed, but let's call it anyway
        graphic.Rebuild(CanvasUpdate.PreRender);

        // Determine effective alpha from CanvasGroups (probably not how Unity does this)
        float effectiveAlpha = graphic.GetComponentsInParent<CanvasGroup>()
            .Aggregate(1f, (alpha, group) => alpha * group.alpha);

        var material = new Material(graphic.materialForRendering); // material with IMaterialModifiers already applied
        material.SetTexture(MainTexProperty, graphic.mainTexture);
        material.SetColor(ColorProperty, graphic.color * new Color(1f, 1f, 1f, effectiveAlpha));
        if (graphic is Text)
        {
            // Not sure how/when Unity decides to set _TextureSampleAdd, but Text is
            // the only component I've encountered that needs it
            material.SetVector(TextureSampleAddProperty, new Vector3(1, 1, 1));
        }

        var mesh = new Mesh();
        // Call protected member Graphic.OnPopulateMesh through reflection to
        // populate the mesh. Probably really slow and skipping quite a few steps
        // in Unity's UI render pipeline, but it seems to work
        using (VertexHelper vh = new VertexHelper())
        {
            graphic.GetType().InvokeMember("OnPopulateMesh",
                System.Reflection.BindingFlags.Instance |
                System.Reflection.BindingFlags.InvokeMethod |
                System.Reflection.BindingFlags.NonPublic,
                null, graphic, new object[] { vh });

            vh.FillMesh(mesh);
        }

        buffer.DrawMesh(mesh, graphic.rectTransform.localToWorldMatrix, material);
    }
}

Attach the component to the camera you’re using for RenderToCubemap and you should be good to go.

Edit: updated to work with CanvasGroup alpha

3 Likes

Not working anymore with Unity 2018.2 any temporary fixes?

I just did a quick test with 2018.2 and it’s still working here. What kind of components are you using in your UI? I only needed the bare basics for my purposes (basically only Text and Image), so it’s possible this code never worked with your UI, even before 2018.2. For example, I know TextMesh Pro components don’t work right now, but I’m not using those in my project so I haven’t looked into why that is.

I am using the same with Unity Recorder to record a 360 video which uses targetCamera.RenderToCubemap(m_Cubemap1, 63, Camera.MonoOrStereoscopicEye.Left); to capture the cubemap in Camera360Input.cs but I am unable to get UI in the output. I am only using basic UI Text and Image.

I’m not sure what it could be then. It’s possible the images are being rendered but at the wrong scale/position; like I said it’s a very hacky solution. I attached my working test project in case you want to investigate further.

3608343–293242–ManualUIRenderer2018Test.zip (22.1 KB)

Thanks got it working in a fresh project.

In order to get the text to appear using these scripts, I had to change

 this.targetCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);

to:

this.targetCamera.AddCommandBuffer(CameraEvent.AfterEverything, commandBuffer);

After that, text appeared! :slight_smile: Thanks so much!

Has anyone here reported a bug about this? If yes, could you please post the case number here?
Also tagging @phil_lira for URP support of this and @LeonhardP as this is a problem in current alpha/beta still.

Doesn’t work with TextMeshPro UI

Thanks for the script… I tested it U2021.1.16f1 and works for normal UI text but not TMP Pro. TMP Pro readers a solid block. So this is perfect for rendering normal text.

Result:
The red rectangle is TMP Pro
The Top text above the Cube is normal UI Text

7394654–903110–equirectangular.zip (2.77 MB)

Would love it if Unity can fix this because it is something I intend to use.

Does any know if any of the commercial assets work properly with 360 renders?

If anyone stumbles across this looking for a fix for TextMeshProUGUI elements (Unsure about TextMeshPro mesh text) I was able to modify Mnstrspeed’s script with the following to get it to render correctly:

var mesh = new Mesh();

// Call protected member Graphic.OnPopulateMesh through reflection to
// populate the mesh. Probably really slow and skipping quite a few steps
// in Unity's UI render pipeline, but it seems to work
using (VertexHelper vh = new VertexHelper())
{
graphic.GetType().InvokeMember("OnPopulateMesh",
System.Reflection.BindingFlags.Instance |
System.Reflection.BindingFlags.InvokeMethod |
System.Reflection.BindingFlags.NonPublic,
null, graphic, new object[ ] { vh });

vh.FillMesh(mesh);
}

if (graphic is TextMeshProUGUI)
{
var textMeshProUGUI = graphic as TextMeshProUGUI;
mesh = textMeshProUGUI.mesh;
material = textMeshProUGUI.fontMaterial;
}

buffer.DrawMesh(mesh, graphic.rectTransform.localToWorldMatrix, material);
2 Likes

UI Buttons that are tinted black are showing up white… Perhaps the button tints aren’t rendered?

Edit:

For anyone else running into this issue, I Modfied Mnstrspeed’s script with jumpgate_lewis’s modification and then further modified it to resolve the above, I did also have to make a second script to keep track of UnityEngine.UI.Selectables selection states, because their selection state is protected, and i didn’t want to replace button with a class that inherits from button or selectable.

My SelectableStateTracker:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;

/// <summary>
/// Unitys default buttons in the normal UI  has selectionstate as a protected variable, can't check it to work around tint not showin in video correctly. so alternatively i track it here.
/// </summary>
public class SelectableStateTracker : MonoBehaviour, IPointerDownHandler,IPointerEnterHandler,IPointerExitHandler,IPointerUpHandler,ISelectHandler,IDeselectHandler
{

    private bool hovered = false;
    private bool pressed = false;
    private bool selected = false;
    public Selectable selectable;

    private void Awake()
    {
        selectable = GetComponent<Selectable>();

    }

    public enum SelectionState {highlighted,normal,pressed,disabled,selected};

    public SelectionState CurrentSelectionState = SelectionState.normal;

    public void OnPointerDown(PointerEventData eventData)
    {
        pressed = true;
    }

    public void OnPointerEnter(PointerEventData eventData)
    {
        hovered = true;
    
    }

    public void OnPointerExit(PointerEventData eventData)
    {
        hovered = false;
    
    }

    public void OnPointerUp(PointerEventData eventData)
    {
        pressed = false;

    }

    public SelectionState evaluate()
    {
        if (!selectable.interactable)
        {
            return SelectionState.disabled;
        }
      

        return (hovered, pressed,selected) switch
        {

            (false, false,false) => SelectionState.normal,
            (true, false, false) => SelectionState.highlighted,
            (false, true, false) => SelectionState.pressed,
            (false, true, true) => SelectionState.pressed,
            (true, true, false) => SelectionState.pressed,
            (true, true, true) => SelectionState.pressed,
            _ => SelectionState.selected
        };
    }

    public void OnSelect(BaseEventData eventData)
    {
        selected = true;
    }

    public void OnDeselect(BaseEventData eventData)
    {
        selected = false;
    }
}

And The Modified version of the ManualUIRenderer

using System.Linq;
using TMPro;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.Rendering;
using UnityEngine.UI;

/// <summary>
/// Component that manually renders world-space UI Graphics to a Camera. Intended for
/// use with Camera.RenderToCubemap which seems to ignore the built-in UI system
/// </summary>
[RequireComponent(typeof(Camera))]
public class ManualUIRenderer : MonoBehaviour
{
    private static readonly int MainTexProperty = Shader.PropertyToID("_MainTex");
    private static readonly int TextureSampleAddProperty = Shader.PropertyToID("_TextureSampleAdd");
    private static readonly int ColorProperty = Shader.PropertyToID("_Color");

    private Camera targetCamera;
    private CommandBuffer commandBuffer;

   

    private void Awake()
    {
        this.targetCamera = this.GetComponent<Camera>();

        this.commandBuffer = new CommandBuffer();
        this.commandBuffer.name = "Manual UI rendering";
        this.targetCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);
        //this.targetCamera.AddCommandBuffer(CameraEvent.AfterEverything, commandBuffer);

       

    }

    private void OnDisable()
    {
        this.commandBuffer.Clear();
    }

    private void Update()
    {
        this.commandBuffer.Clear();
        AddUiDrawingCommands(this.targetCamera, this.commandBuffer);
    }

    private static void AddUiDrawingCommands(Camera cam, CommandBuffer buffer)
    {
        // Root canvases ordered by screen space depth
        var rootCanvases = FindObjectsOfType<Canvas>()
            .Where(canvas => canvas.isRootCanvas && canvas.renderMode == RenderMode.WorldSpace)
            .OrderByDescending(canvas => cam.WorldToScreenPoint(canvas.transform.position).z);

        foreach (var canvas in rootCanvases)
        {
            // Graphics after culling sorted by depth
            var graphics = canvas.GetComponentsInChildren<Graphic>()
                .Where(graphic => TestCullingMask(graphic, cam.cullingMask))
                .OrderBy(graphic => graphic.depth);

            foreach (Graphic graphic in graphics)
            {
                AddGraphicDrawingCommands(graphic, buffer);
            }
        }
    }

    private static bool TestCullingMask(Graphic graphic, int cullingMask)
    {
        return ((1 << graphic.gameObject.layer) & cullingMask) != 0;
    }

    private static void AddGraphicDrawingCommands(Graphic graphic, CommandBuffer buffer)
    {
        // Probably not needed, but let's call it anyway
        graphic.Rebuild(CanvasUpdate.PreRender);

        // Determine effective alpha from CanvasGroups (probably not how Unity does this)
        float effectiveAlpha = graphic.GetComponentsInParent<CanvasGroup>()
            .Aggregate(1f, (alpha, group) => alpha * group.alpha);

        var material = new Material(graphic.materialForRendering); // material with IMaterialModifiers already applied
        material.SetTexture(MainTexProperty, graphic.mainTexture);
        material.SetColor(ColorProperty, graphic.color * new Color(1f, 1f, 1f, effectiveAlpha));

        if(graphic.GetComponent<SelectableStateTracker>() != null)
        {
            SelectableStateTracker s = graphic.GetComponent<SelectableStateTracker>();
            if (s.selectable.transition == Selectable.Transition.ColorTint)
            {
                SelectableStateTracker.SelectionState state = s.evaluate();

                Color c = (state) switch
                {
                    SelectableStateTracker.SelectionState.highlighted => s.selectable.colors.highlightedColor,
                    SelectableStateTracker.SelectionState.normal => s.selectable.colors.normalColor,
                    SelectableStateTracker.SelectionState.pressed => s.selectable.colors.pressedColor,
                    SelectableStateTracker.SelectionState.disabled => s.selectable.colors.disabledColor,
                    SelectableStateTracker.SelectionState.selected => s.selectable.colors.selectedColor,
                    _ => s.selectable.colors.normalColor
                };


                material.SetColor(ColorProperty,c * new Color(1f, 1f, 1f, effectiveAlpha));
            }

        }

        if (graphic is Text)
        {
            // Not sure how/when Unity decides to set _TextureSampleAdd, but Text is
            // the only component I've encountered that needs it
            material.SetVector(TextureSampleAddProperty, new Vector3(1, 1, 1));
        }

        var mesh = new Mesh();



        // Call protected member Graphic.OnPopulateMesh through reflection to

        // populate the mesh. Probably really slow and skipping quite a few steps

        // in Unity's UI render pipeline, but it seems to work

        using (VertexHelper vh = new VertexHelper())

        {

            graphic.GetType().InvokeMember("OnPopulateMesh",

                System.Reflection.BindingFlags.Instance |

                System.Reflection.BindingFlags.InvokeMethod |

                System.Reflection.BindingFlags.NonPublic,

                null, graphic, new object[] { vh });



            vh.FillMesh(mesh);

        }



        if (graphic is TextMeshProUGUI)

        {

            var textMeshProUGUI = graphic as TextMeshProUGUI;

            mesh = textMeshProUGUI.mesh;

            material = textMeshProUGUI.fontMaterial;

        }



        buffer.DrawMesh(mesh, graphic.rectTransform.localToWorldMatrix, material);

    }
}