Rendering part of a texture inside of a drawn shape?

Unity Engine
,
1

Hi all,

I’m working on a 2D platformer where the player can switch between 2 worlds. Objects in the scene are on separate layers, corresponding to the world they’re supposed to be in. At the press of a button, the player’s layer changes.

The feature i’m having trouble implementing is for the player to be able to draw on the screen and when the player draws a closed shape, ie. a circle, the area inside the closed shape switches to the other world.

Here’s what I’ve done so far:
I have a Line gameobject with a Line Renderer component and a script that works well enough for now that has the drawn line disappear immediately if it’s not a closed shape, and it lingers for 5 seconds if it’s a closed shape.
I’ve created a custom shader which has 2 texture slots for 2 worlds and is applied to all objects in the scene, and the player can change which texture is shown with material.SetFloat depending on the world you’re in (SetFloat changes between 0 and 1 and changes the texture based on that)

How would I go about changing the float value inside the drawn shape? Or is there a better way to approach this?
Thanks.

2

Are you trying to mask the layer contents with the drawn shape?

There are several ways to approach this, but for your case I think you’d have to produce a black & white texture from the closed shape and then use this in your custom shader to mask the content.

The fastest way to build such a texture is not to draw it on a CPU, but instead to build a polygon (in its own layer) and use a new, layer-specific camera to render this polygon to a render texture. This is your mask.

To build a polygon you need a library capable of mesh Delaunay triangulation, that’s the term for optimally producing a reasonable-looking mesh out of a series of points (which need not be in order), regardless of whether the polygon is convex or concave.

There is a lot of stuff you need to unpack to make this happen, but there you go, as a high level breakdown. I’m sure other people can point you to more specific paths you could take. Or even entirely different solutions. I’m not claiming this is the best one, but I’d argue it’s one of the best performing, even for a mobile game. If done right, of course.

3

Hey orionsyndrome, thanks for answering.

I’m only a few months into Unity so trying to implement this has been an oscillation between getting excited thinking I’m on the right track and getting depressed realizing some of these processes are going over my head.

The approach I’ve decided to go for is to indeed create a mask of the drawn shape, like you suggested, and then use stencil buffer in my shader to render the mask. Here’s the relevant code for creating the mask of the closed shape:

private IEnumerator Linger()
    {
        Vector3 center = GetCenter();

        GameObject portal = new GameObject("Portal");
        portal.transform.position = center;
        portal.AddComponent<MeshFilter>();
        MeshRenderer meshRenderer = portal.AddComponent<MeshRenderer>();

        Mesh mesh = CreateMesh();
        portal.GetComponent<MeshFilter>().mesh = mesh;

        Material material = new Material(Shader.Find("Standard"));
        meshRenderer.material = material;
        material.SetInt("_StencilComp", (int)UnityEngine.Rendering.CompareFunction.Always);
        material.SetInt("_StencilOp", (int)UnityEngine.Rendering.StencilOp.Replace);
        material.SetInt("_StencilWriteMask", 0xFF);
        material.SetInt("_StencilRef", 1);

        yield return new WaitForSeconds(lingerTime);
        line.positionCount = 0;
        Destroy(portal);
    }

        private Vector3 GetCenter()
    {
        Vector3 center = Vector3.zero;
        for (int i = 0; i < line.positionCount; i++)
        {
            center += line.GetPosition(i);
        }
        center /= line.positionCount;
        return center;
    }

    private Mesh CreateMesh()
    {
        Mesh mesh = new Mesh();
        Vector3[] vertices = new Vector3[line.positionCount];
        int[] triangles = new int[(line.positionCount - 2) * 3];
        for (int i = 0; i < line.positionCount; i++)
        {
            vertices[i] = line.GetPosition(i);
        }
        for (int i = 0; i < line.positionCount - 2; i++)
        {
            triangles[i * 3] = 0;
            triangles[i * 3 + 1] = i + 1;
            triangles[i * 3 + 2] = i + 2;
        }

        mesh.vertices = vertices;
        mesh.triangles = triangles;
        mesh.RecalculateNormals();
        mesh.RecalculateBounds();
        Vector3 meshSize = mesh.bounds.size;

        Vector3 center = GetCenter();
        for (int i = 0; i < vertices.Length; i++)
        {
            vertices[i] -= center;
        }
        mesh.vertices = vertices;
        mesh.bounds = new Bounds(center, Vector3.one * meshSize.magnitude);


        return mesh;
    }

I create a gameObject called Portal depending on the line positions of the drawn object and it works well. I’m having trouble wrapping my head around how the stencil buffer would go about rendering that part of the screen to be opposite relative to the rest of the screen.

Usually, questions I had so far took a bit of googling but the answers were there. This implementation seems specific enough that I can’t find anyone discussing it.

I have been at this only a couple of days but it does seem sufficiently advanced that I might just cut it for now and try to tackle it when I’m more experienced with Unity, however, if I’m on the right track here I’d very much like to see it through.

4

Hey, you’re on the right path! Yes stencil is also a good way to do it, although it’s quite unintuitive for me, I too have to google it every time it pops up, so I tend to avoid it unless it’s obviously something only a stencil would do.

Regarding the polygon, are you building a fan mesh? You know your path stroke, I guess if you demand the player to produce a path in one mouse swoop, that’s just fine, much simpler than what I proposed.

You can also avoid having to compute bounds yourself, just use SetVertices instead (it will auto compute the bounds).

For the center point you can just sum the vertices up (in the for loop), and divide that by line.positionCount. That’s the standard average. Oh wait, you’re already doing it with GetCenter.

You also don’t need RecalculateNormals because you know all of them just point against the camera.

You can do something along these lines

static List<Vector3> _verts = new List<Vector3>();
static List<Vector3> _normals = new List<Vector3>();
static List<int> _tris = new List<int>();

private Mesh CreateMesh()
{
  _verts.Clear();
  _normals.Clear();
  _tris.Clear();

  int count = line.positionCount;
  var mesh = new Mesh();
  var sum = Vector3.zero;

  for (int i = 0; i < count; i++)
  {
    var p = line.GetPosition(i);
    _verts.Add(p);
    _normals.Add(Vector3.back);
    sum += p;
  }

  var centerIndex = _verts.Count;
  _verts.Add(sum / count);
  _normals.Add(Vector3.back);

  for (int i = 0; i < count - 2; i++)
  {
    _tris.Add(centerIndex);
    _tris.Add(i);
    _tris.Add(i + 1);
  }

  mesh.SetVertices(_verts);
  mesh.SetNormals(_normals);
  mesh.SetTriangles(_tris, 0);

  return mesh;
}

I apologize for errors or typos.

5

Oh I forgot to mention that you have an issue when the player draws a concave shape. You can’t draw all concave shapes with the fan method. For example if you begin the path clockwise, and then switch to ccw, and return to cw, you will inevitably have some inverted triangles.

You can detect this in the triangle building loop if you compute the normal for each triangle. If your triangle is A,B,C you can compute the normal by doing a cross between (B-A) and (C-A) edge vectors. But in this case you only care about the sign itself. All of your triangles need to have the same sign.

You can turn this test into a local function inside CreateMesh, which should look like this

static int triSign(Vector3 a, Vector3 b, Vector3 c, Vector3 normal)
  => Vector3.Dot(Vector3.Cross(b - a, c - a), normal) > 0f? 1 : -1;

Then, in the triangle building part, you can simply do

  var sign = 0;

  for (int i = 0; i < count - 2; i++)
  {
    _tris.Add(centerIndex);
    _tris.Add(i);
    _tris.Add(i + 1);

    if(i == 0)
    {
      sign = triSign(_verts[centerIndex], _verts[i], _verts[i+1], Vector3.back);

    }
    else
    {
      if(sign != triSign(_verts[centerIndex], _verts[i], _verts[i+1], Vector3.back))
      {
        Debug.Log("Backfacing triangle encountered");
        return null; // signals failure
      }

    }
  }
6

Thanks for refactoring my code! Much appreciated. What I’m having issues with is getting the stencil buffer to do its rendering within the circle

I have the stencil added in my custom shader:

Shader "Custom/NewSurfaceShader" {
    Properties{
        _Color("Color", Color) = (1,1,1,1)
        _MainTex("Albedo (RGB)", 2D) = "white" {}
        _Glossiness("Smoothness", Range(0,1)) = 0.5
        _Metallic("Metallic", Range(0,1)) = 0.0
        _PBRReality("PBR Reality", 2D) = "white" {}
        _PBRRealm("PBR Realm", 2D) = "white" {}
        _StencilTex("Stencil Texture", 2D) = "white" {}
    }

    SubShader{
        Tags {"RenderType" = "Opaque" "Queue" = "Geometry"}
        LOD 100

        Pass{
            Stencil {
                Ref 1
                Comp always
                Pass replace
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"

            sampler2D _PBRReality;
            sampler2D _PBRRealm;
            sampler2D _StencilTex;

            float _PlayerLayer;

            struct appdata {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f {
                float2 uv : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };

            v2f vert(appdata v) {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = v.uv;
                return o;
            }

            fixed4 frag(v2f i) : SV_Target{
                fixed4 finalColor;
                if (_PlayerLayer == 0.0) {
                    finalColor = tex2D(_PBRReality, i.uv);
                } else {
                    finalColor = tex2D(_PBRRealm, i.uv);
                }

                fixed stencilValue = tex2D(_StencilTex, i.uv).r;
                finalColor.a *= stencilValue;

                return finalColor;
            }
            ENDCG
        }
    }

    FallBack "Diffuse"
}

And then referenced in the Linger coroutine:

private IEnumerator Linger()
    {
        Vector3 center = GetCenter();

        GameObject portal = new GameObject("Portal");
        portal.transform.position = center;
        portal.AddComponent<MeshFilter>();
        MeshRenderer meshRenderer = portal.AddComponent<MeshRenderer>();

        Mesh mesh = CreateMesh();
        portal.GetComponent<MeshFilter>().mesh = mesh;

        Material material = new Material(Shader.Find("Custom/NewSurfaceShader"));
        meshRenderer.material = material;

        // Set the initial stencil parameters for the portal
        material.SetInt("_StencilComp", (int)UnityEngine.Rendering.CompareFunction.Always);
        material.SetInt("_StencilOp", (int)UnityEngine.Rendering.StencilOp.Replace);
        material.SetInt("_StencilWriteMask", 0xFF);
        material.SetInt("_StencilRef", 1);

        // Set the player layer float based on the current layer of the player
        int playerLayer = player.layer;
        float playerLayerFloat = playerLayer == LayerMask.NameToLayer("Realm") ? 1.0f : 0.0f;
        material.SetFloat("_PlayerLayer", playerLayerFloat);

        yield return new WaitForSeconds(lingerTime);

        // Set stencil comparison function to equal and stencil reference to 1
        material.SetInt("_StencilComp", (int)UnityEngine.Rendering.CompareFunction.Equal);
        material.SetInt("_StencilRef", 1);

        // Destroy the portal and clear the line renderer
        line.positionCount = 0;
        Destroy(portal);
    }

The Portal gameobject is created along the lines of the drawn shape, but nothing changes as far as rendering it goes. Is there something in the camera settings that needs to be adjusted to render the stencil mask? Also, as far as I understand, stencil buffer is a way for part of one camera to be rendered differently, so this method doesn’t require a separate camera to render it, correct?
I’m on standard 2D project, do I need URP for this to work?

7

No, stencils work with built-in rendering just fine. In fact, ShaderLab features you’re relying upon were developed for built-in.

The way it works is that the stencil mask acts as a “guardian” for whatever the shader is normally doing. It literally controls whether the fragment shader will be called or not. To automate this you have a stencil buffer at a disposal, to which you can write, only for these values to be compared by another shader which is supposed to do some other work.

If you have scene A and scene B, and you wish to create a “portal” that can show scene B, constrained to some polygon, yet superimposed on top of scene A, I believe you’d need 2 cameras (each for its world/layer to render) and 2 shaders (one maskable by the stencil buffer, and the other to actually write the mask, which you put on your polygon).

Don’t forget you can make your stencil parameters externally configurable, if you do

Properties {
  _RefValue ("Stencil ID [0;255]", Int) = 0
}

...

Stencil {
  Ref [_RefValue]
  ... // other test parameters can be also configured like this
}

Check out this git repo for more details (Id is float there for some reason, I think this is an error; but anyway, trust them more than you trust me)

Here’s a video depicting how to achieve this

  • Turn on the subtitles!
  • It’s URP but the workflow shown here is through ShaderLab, so it should be the same as with built-in
  • (There is another SRP-specific workflow you can see in some other stencil tutorials)

You probably need also to configure your two cameras to render on top of each other, where the second render is superimposed. You need to learn this independently of this problem.

8

Hey, thanks for providing links! They were very helpful. With them in hand, I decided to take a step (or 10) back and try to replicate what I’m trying to do in the simplest way possible, just to figure out how stencil buffer works… And I did! Sort of…

[Album] imgur.com

My original shader has 2 texture slots, and what needs to happen is part of the currently unrendered texture needs to be rendered within the second plane that serves as a mask. Right now the mask simply renders whatever is in its shaders texture slot or just white if it’s empty.

As I’m writing this and re-reading your latest reply, I feel like it’s coming to me why you suggested 2 cameras. The way it currently works is that the player when switching the world he is in, he simply switches which of the 2 textures the shader will render. So I guess, in this context, it’s not possible (or at the very least it’ll take a lot more headache) to get the snippet of the currently unrendered texture.

So with 2 cameras I can have both textures rendered, each to their respective camera and then use the stencil buffer to get part of the opposite world rendered…

Or maybe not… This has been the bane of my existence the past week, I’ll try to play with it some more tomorrow before admitting defeat.

8980945--1235572--1211689-ac246178e591fff2c3b70535f2e0963b.jpg