Add Shader graphs effect to the AR camera feed live

Archived Forums Questions & Answers
, ,
1

I am trying to add a distortion effect that is created with shader graph to the AR camera feed live. Currently, the raw image shows nothing and it is fully transparent.
I created a raw image on canvas
I implement a render texture to the texture of this raw image.(Depth Stencil Format: D16_UNORM)
I create a script to get the living AR camera feed and add shader graphs.(added to AR camera. Camera > rendering> post processing> checked. AR camera background component > use custom material > checked. added Volume > profile > new AR volume profile)
The following is code of it

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation; // Import AR Foundation namespace
using UnityEngine.XR.ARSubsystems; // Required for accessing camera image

[RequireComponent(typeof(ARCameraManager))]
public class ARCameraFeedToRenderTexture : MonoBehaviour
{
    public RenderTexture targetRenderTexture;
    private ARCameraManager arCameraManager;
    public Material distortionMaterial; // Your custom material
    // Materials for handling iOS textures
    public Material iOSMaterial; // This material should be able to handle Y and CbCr textures

    void Awake()
    {
        arCameraManager = GetComponent<ARCameraManager>();
    }

    void OnEnable()
    {
        arCameraManager.frameReceived += OnCameraFrameReceived;
    }

    void OnDisable()
    {
        arCameraManager.frameReceived -= OnCameraFrameReceived;
    }

    void OnCameraFrameReceived(ARCameraFrameEventArgs args)
    {
        // Check if we are running on iOS or Android
        if (Application.platform == RuntimePlatform.IPhonePlayer && args.textures.Count >= 2)
        {
            // For iOS
            // Assuming texture[0] is the Y component and texture[1] is the CbCr component
            Texture2D textureY = args.textures[0];
            Texture2D textureCbCr = args.textures[1];

            // Set textures on the iOS material
            iOSMaterial.SetTexture("_textureY", textureY);
            iOSMaterial.SetTexture("_textureCbCr", textureCbCr);

            // Use the iOS material to blit to the render texture
            Graphics.Blit(null, targetRenderTexture, iOSMaterial);
        }
        else if (args.textures.Count > 0)
        {
            // For Android
            Graphics.Blit(null, targetRenderTexture, distortionMaterial);
        }
    }

}

i am using Unity 2021.3.5f1.
AR foundation version 4.2.10
project settings > quality > render pipeline asset > URP-forward-render(universal render pipeline asset).
project settings > Graphics > scriptable render pipeline settings > URP-forward-render(universal render pipeline asset)
URP-forward-render-renderer
added renderer feature > AR Background renderer feature.

do monkey dance to solve it

 –

1 Answer

1
2

It seems like you’ve got the right pieces in place. You’re capturing the AR camera feed and trying to apply your custom shader effect using Unity’s AR Foundation.

Make sure your shader graph is correctly set up to handle the input from the AR camera and output the desired effect. Sometimes the issue might be with the shader graph not being properly configured to work with the AR data.
In your script, you’re handling the texture differently for iOS and Android. Ensure that the materials (distortionMaterial and iOSMaterial) are correctly processing the textures from the camera feed. Also, verify that your AR Foundation and other Unity settings are correctly configured to support shader effects on the camera feed.

If all the technical parts are set right, and you’re still seeing a transparent image, there might be an issue with how the textures from the camera are being passed to the shader or how the shader is processing them.