Okay, instead of trying to figure out what’s wrong with the math you have, lets take a step back and look at what you’re doing.
Every single pixels you’re iterating over every single triangle’s position to try and find one you’re coplanar with, in part to figure out which triangle the current pixel you’re rendering is a part of, and eventually to get the barycentric coordinates.
Don’t do that. The shader already knows which triangle it’s rendering.
float4 frag (v2f i, <strong>*uint pID : SV_PrimitiveID*</strong>) : SV_Target
The SV_PrimitiveID semantic stores the current triangle index being rendered and can be accessed in the fragment shader. That index should exactly match the mesh’s triangle list as you see it from script. So instead of looping over every triangle’s vertices to find which one you’re on, you just need to do this:
int pA = saTriIndex[pID].x;
int pB = saTriIndex[pID].y;
int pC = saTriIndex[pID].z;
However even this isn’t really the best way to go about this. Since all you really want is the barycentric coordinates, the most straightforward solution in Unity is to use a geometry shader. There are plenty of examples for that, however I wouldn’t necessarily recommend that option, as there’s an even cheaper solution. Just store the barycentric coordinates in the vertices. Assuming you’re generating your own mesh, the simple way is to have each triangle’s vertices be unique and store (1,0), (0,1), and (0,0) in the three vertices of the triangle, either in an extra UV channel or the vertex color. The resulting interpolated value can be used to determine the barycentric coordinate. The geometry shader technique works exactly the same way, only these values are injected into the mesh at render time by the shader rather than being stored in the mesh.
The slightly harder way is to not break up the triangles and just make sure each triangle’s vertices have only one of those three values. I know of at least two assets on the store that come with utilities to do just that.
https://assetstore.unity.com/packages/vfx/shaders/wireframe-shader-the-amazing-wireframe-shader-18794
https://assetstore.unity.com/packages/tools/terrain/megasplat-76166
That second asset, MegaSplat, is specifically designed to do exactly what you’re doing in a hyper efficient way.
However all of this might still be completely unnecessary depending on what exactly you need. If you only need a total of 4 or 5 texture layers you could just use the vertex colors as a mask. I’m going to guess based on how you were trying to calculate the barycentric coordinates that every pixel is already sampling all of the textures all of the time anyway.