Dual mask stencil issue

I’m trying to get something like this in my Unity game

Let me explain myself. My game is about changing between two overlapped worlds, you can see a demo here:

Here I use a sprite mask, when the player is inside that it onlly takes into account certain colliders.

Now, this have en issue, I want to have another “portal” moving in the world, something like this:

Now, this have an issue, imagine that the player changes dimension (it just grows a mask that always follow the player), here the user could expect 2 things. The first one is that both masks “fuse”, the second one is that the other mask gets “inverted” and shows always the opposite world. Ths is what I want. The issue is that I can’t get that behaviour with sprte masks nor with the stencil buffer directly.

I did a quiick prototype in Love2D that works, I don’t know how to replicate that in Unity so I will tell you now what I did in Love

When rendernig, I use a canvas (equivalent to uniity render texture), I render all masks of type 1 there, then I render that texture to the stenciil buffer (discarding transparent pixels as they are not part of the mask). When rendering to stencl, I use the “increment” function. Then I do the same with the other mask type. Now, the stencil will be 1 where there is a mask of type 1, 2 where they overlap with the type 2 and 0 where theere are no masks. Now that I have the stencli the rest is easy, I render the first world normallly, with its post processsing and stuff. Then, I set the stencil test to (“notequall” 1) and with that I render the second world on top. This works perfectly, I have masks that “fuse” and other ones that “invert” when overlapping. This is the result:

Good but… How do I do that in Unity? I think I now that I can render the masks to one render texture each using two cameras, but I don’t know how to combiine then. I have more render issues because I need things to render in a really specific way, but I think that once I solve this, I wiill be able to figure out the rest by myself. Also, in Love2D I could get away with only one canvas/render texture because I had control over the whole render pipelline. Here I will probably need two cameras and two render textures. Also, the stencil should be computed at the end of the render process, before anything else, then I can just use that stenciil buffer whenever I want.

So, Where Am I now to sum up? I can render the two mask types to two render texture using 2 cameras. But I don’t know how to combine that in the stencil and apply it. I fear I will need another 2 cameras one to render the first world another one for the second one, but I stll don’t know how to tell Unty “alright, render the masks, now render the first render texture to stencil, now render the second one, ,alrght now render world 1, alright now enable stencil, finally render the other world with the stencilii applied…”. I also want to ignore the stencli on certain objects temporarlly to draw on top of everythnigi witihi low opacity (like in the first video but that’s a horrible hack)), but I’ll leave that render isssue for when I manage to get this one solved…

If you wonder how it’s working now, it’s awful, II’m using 2 cameras but now to render textures, I also have 2 player syncronized so they render in both world correctly as each world has its own liights and stuff. This could work with 1 camera but I use 2 because of Unity lights, anyways it seems that havnig 2 would be a requeriiment sooner or later.

Well, this is all, I hope I explained myself well. If I manage to fix before someone post a solution I’ll write iit here, otherwise you’ll see below a “marked as answer” comment or something like that

1 Like