How to mask any sprite/particles with feathering?

I know sprite mask is limited and the stencil pass of unity is binary so it cannot work with transparency

So that leaves it up to a shader. The major issue I’ve faced is: How to make the shader behave with unity world coordinates, so that it carefully can detect the world position of each pixel and the intersections between the objects with the different materials (material that acts as a mask), (material that gets masked), with option to visible inside or outside the mask.

here is my desired effect:
7338058--892060--Untitled-1.png
in this case a particle system is used to mask a blue square.
So this would be, particle system(MASK) and blue square(visible inside mask)

another example:


the multicolor squares should act as a mask (have the mask shader), and the character with the shader (get masked), with the option visibile outside mask.

There should be the many of these objects on the scene, moving freely in world space, and also make it so that theres also an option to ignore certain objects. ( For example if I have sprite A and sprite B, mask A will probably affect both sprites, but i need to be able to set it in a way that mask A only masks sprite A and not sprite B, instead sprite B should only be responsive to mask B.)

Is this possible to achieve?

Man I tried explaining this to you but a book doesn’t help, because but this is immensely hard to do in Unity.

You basically want render texture compositing (edit: you build a mask with another camera essentially), but this is so broken in the built-in rendering that it took me more than a week until I reached a point I would call it decent. But the API headache, the lack of documentation, and the non-trivial issues with how Unity deals with texture Y-flipping, as well as having to build a custom compositing shader, is really a fantastic cost to bear.

I can only hope that this is easier and less broken with URP. Hopefully someone else will provide an insight into that, because I’m not yet fully versed with URP.

1 Like

I know nothing about how unity does rendering but i feel like there must be a crucial design flaw that prevents masking with alpha 0.01-0.99, i feel like sprite mask has gone this far without allowing feathering because they literally cannot add it without reworking some kind of pillar of code. All i know is that there is a stencil pass and the pixel is either full opaque or full transparent and it doesnt allow for anything in between

I thought shaders could help but I always fail when it comes time to work with world space and interacting with other objects. I also see that there are assets on the asset store that advertise sprite mask with feathering but it always breaks when you try using multiple masks, and trying to make it apply only to the things you want

Actually that part is covered by the shaders, and has almost nothing to do with Unity, and it works as expected. However, Unity provides the rendering system, the so called pipeline, and the way you’re supposed to interconnect everything. And this is what is ridiculous, that such obvious things, like image compositing, multi-shader blitting, dynamic recombination of multiple image sources, are extremely hard to achieve. It’s not because “you can’t do it”, no you can, but I bet you’ll lose a good portion of your hair by the time you get it right.

Again, this experience of mine is based strictly on the built-in rendering. Things like this should be the reason why they introduced SRPs, so this could be a different experience altogether in URP/HDRP. At least I really really hope so. So take my advice with a grain of salt.