Cull Pixels Inside Overlapping Primitive

Re-posting this here as I originally posted in HDRP forum and have 0 responses as of yet.
link: Cull Pixels Inside Overlapping Primitive

Hello Unity Community,

Using HDRP 2021.2.0b15.3666

Is it possible to cull pixels that fall inside of another overlapping primitive. So you have Sphere A with culling Shader on it along with a Cullme script, then Sphere B touches/overlaps a portion of Sphere A. All pixels from Sphere A, that fall inside Sphere B, then get culled off of Sphere A. Sphere B is transparent.

I am very determined to continue working inside HDRP and really could use a nudge in the right direction with this. I am trying to recreate the L4D effect of culling a mesh and then showing another mesh hidden underneath (insides).

I got this half working with a CGPROGRAM shader file copied from another user (sorry I don’t recall the source). Tried adding that to a Custom Node (String and File) in Shader Graph and hitting a wall. Do I need to sign up for Unity HDRP Shader Classes or is this something someone here can help me with? Is that even a thing? If I wanted to pay someone to make this as an HDRP LIT shader that can be culled, where would I post for that?

Attached is a zip file of a short video showing the CGPROGRAM shader in action, in this HDRP project.

Any help is appreciated. :slight_smile:

     Shader "CGShader5"
         {
             SubShader
             {
                 Pass
                 {
                     /*Cull Off*/
     
                     CGPROGRAM
     
                     #pragma vertex vert
                     #pragma fragment frag
     
                     uniform float4x4 _CullerSpace;
     
                     struct vout
                     {
                         float4 pos : SV_POSITION;
                         float4 worldpos : TEXCOORD0;
                         float4 localpos : TEXCOORD1;
                         float4 cullerpos : TEXCOORD2;
                     };
     
                     vout vert(float4 vertex : POSITION)
                     {
                         vout _out;
                         _out.pos = UnityObjectToClipPos(vertex);
                         //_out.pos = mul(Unity_Matrix_VP, Unity_ObjectToWorld), (vertex);
                         _out.localpos = vertex;
                         _out.worldpos = mul(unity_ObjectToWorld, vertex);
                         _out.cullerpos = mul(_CullerSpace, _out.worldpos);
                         return _out;
                     }
     
                     float4 frag(vout _in) : COLOR
                     {
                         /*return float4(_in.cullerpos.x, _in.cullerpos.y, 0, 1);*/
     
                         if (_in.cullerpos.x > -0.5 && _in.cullerpos.x < 0.5 &&
                             _in.cullerpos.y > -0.5 && _in.cullerpos.y < 0.5 &&
                             _in.cullerpos.z > -0.5 && _in.cullerpos.z < 0.5)
                             discard;
     
                         return float4(0, 1, 0, 1);
                     }
     
                     ENDCG
                 }
             }
         }
 using UnityEngine;
[ExecuteInEditMode]
public class CullMe : MonoBehaviour
{
     public Transform culler;
     void Update()
     {
         if (!culler)
             return;
         var renderer = GetComponent<Renderer>();
         if (!renderer)
             return;
         var material = renderer.sharedMaterial;
         material.SetMatrix("_CullerSpace", culler.worldToLocalMatrix);

From the time I posted this, I have been experimenting with different approaches. One that has gotten me the closest results is from here:

link: Unity3d Cross Section Shader Using Shader Graph | by Abdullah Aldandarawy | codeburst

The problem is that I can only get it working on a plane right now. I want to get it working on a sphere so that ONLY INTERSECTING PIXELS get culled, not the entire model. When I swapped in a sphere for the plane, it wasn’t actually doing it based off the intersection but rather the planes position and normal vector.

My thoughts at this point are that I need to figure out the equivalent for the CGSHADER which does exactly what I want. However, I am running into a roadblock with figuring out how to replicate the logic inside Shadergraph. I was thinking that I need to be using a Custom Node in shadergraph and feed in the Intersecting TEXCOORD’s … somehow. Isn’t there a way to just reference this File in the Custom Node and have it work that way?

Does anyone know of any working examples I can look at or if there is a tutorial you know of that can help me along in this pursuit? Anything? I am all for putting in the leg work, I just need a little guidance please. The end goal is to instantiate a transparent mesh onto another mesh, and then cull the intersecting pixels.

Taking a Jettely course which specifically goes over how to create/configure the Custom Node and create the HLSL file from scratch to input the values/functions. Hoping to have a rough draft done by tomorrow afternoon and will post the final result then. :slight_smile:

I am convinced that this just isn’t possible. I have tried messing around with replicating the CGSHADER5 functions in ShaderGraph. The closest I have come is to cut the entire mesh at once, not one section of the mesh based on where another object intersects. Another version takes the POSITION node and then try’s to subtract it from the UV node but that shows the seams and I didn’t get the Intersection part to work. Spent three day’s off researching and trying to replicate this effect.

I had already created another effect that uses Forward Rendering and partially achieves the effect I want. The problem is that it creates a new mesh on-top of the existing mesh and just adds more to the scene, rather than have a shader just cull the pixels away. Also I need to figure out how to wrap the top mask to the mesh so it can compensate for non flat surfaces. And how to merge it with existing scars when they overlap eachother.

From here I think I will just table it and when I have more time off I will try to merge the CGSHADER5 code directly into an HDRP shader or maybe URP instead. Looks like the URP code is easier to work with. URP unlit basic shader | Universal RP | 8.2.0 Maybe I will try creating a new project in URP and copy pasta that code into the URP shader and mess around with that.

7944922–1016680–ScarPlacer.7z (7.98 MB)