Blur camera feed (ARCamera)

Unity Engine
,
1

I’m using ARFoundation to do facetracking on iOS with blendshapes. I want to slightly blur the camera feed you see in the background, so your actual face (and environment) is not recognisable.
I know you can use ‘Custom Material’ on the ARCamera component. Which on iOS, it feeds the material 2 textures (textureY and textureCbCr).

Can anybody help out either to write a shader which blurs those textures or do it in another way (renderpass? renderfeature? blit?)

Thanks in advance!

2

did you ever figure out a solution to this? Looking for something similar

3

Nope…

4

It depends on the renderpipeline that you are using.
It is easier with the builtin render pipeline but URP is the current way to go.

We were doing a selective background blur in combination with additional postprocessing so i haven’t tested the shorter method that i will describe.
The tricky part is at which moment you draw the camera image.
With the default behaviour you can choose between before opaques and after opaques. This is just an optimisation to reduce the amount drawn when for example the whole camera is blocked by a 3D object. Since we are not going for high performance anyways with the blur i would suggest just working back to front.

  1. Draw the camera feed before opaques, you might be able to even use the default material for that
  2. Blur the image
  3. Draw the 3D elements

For the blur i went for a separable blur shader with a downsampling step and then a horizontal and a vertical pass.
Here is the render feature setup. (If you are going for the builtin pipeline you have to add the command buffers directly)

public class BlurPass : ScriptableRenderPass
    {
        private string _passName = "Camera Blur Pass";
        public Material blurMaterial;
        public RenderTexture mask; // ignore the mask stuff

        public BlurPass()
        {
            // This needs to be before opaques
            renderPassEvent = RenderPassEvent.AfterRenderingTransparents;
        }
      
        public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
        {
            if (blurMaterial == null) return;
            var renderer = renderingData.cameraData.renderer;
            var cmd = CommandBufferPool.Get(_passName);
            cmd.BeginSample(_passName);
            RenderTexture tmpRT = RenderTexture.GetTemporary(mask.descriptor);
            cmd.Blit(renderer.cameraColorTarget, tmpRT, blurMaterial, 0); // horizontal pass
            cmd.Blit(tmpRT, renderer.cameraColorTarget, blurMaterial, 1); // vertical pass
            cmd.EndSample(_passName);
            context.ExecuteCommandBuffer(cmd);
            CommandBufferPool.Release(cmd);
        }
    }

What this does is basically draw the camera image that is already drawn into the current render target into a smaller temporary texture. During this step we also perform the horizontal blur.
Then in the next operation the vertical blur is performed and drawn back into the render target.

In our case the process was a little different due to the desired postprocessing and the masked blur. The trick there is basically draw the 3D Stuff, copy transparency, do postprocessing and then compose the camera feed with the transparency.

2 Likes