I am currently trying to implement an effect, but my google-fu isn’t strong enough.
I have a shader that is rendered additively. It is transparent, but I don’t want to have the typical artefacts with concave or self-intersecting geometry, so I added a Depth-only pass. This works great as long as the objects are all nicely sortable, but in my case I have objects contained in other objects.
I can mostly fix it by forcing a render order using the RenderQueue, but in some cases, the order is not really deterministic.
Is there a way emulate a depth test in a way that it only applies to the currently rendered object, or reset the depth buffer after an object is drawn?
What you’Re asking is simply not possible. Transparent object / triangles have to be rendered with the painter’s algorithm to get the correct blending between the different parts. Most engines can only sort on a per object basis which is what Unity does as well. However objects can only be sorted based on their pivot position. You can not somehow determine a single depth value for a whole object from the current view position that ensures the correct order. To implement the painter’s algorithm properly it’s not even enough to sort on a per triangle basis as intersections or circular overlaps are edge cases which would make it necessary to split a triangle / polygon into several to be able to sort them correctly.
For most cases this is completely impractical in a real time rendering setup. Keep in mind that the order of objects / triangles would also change depending on the view. You haven’t said what kind of game / application you’re creating. In 2.5D or 2D setups this becomes a bit simpler. However in a full 3D environment there’s no generally valid approach.
The first problem is that the individual triangles of a mesh are rendered in the order they are defind in the mesh. That’s also why self occluding geometry can look so weird depending on the angle you look at it. Because from one point the triangles may be rendered in the right order because the first triangles in the mesh are the furtherst from your view point. So they would follow the painter’s algorithm. However viewing the same object from the otherside would of course reverse the order. Since most more complex imported objects may have an arbitrary triangle order you almost never get the right sorting. This is just one single object. Of course you don’t want any self overlapping so you first do a depth pass so only the closest triangles are rendered.
The next issue is sorting of the individual objects is difficult since you would need to take into account every part of each object in order to sort them properly. For example imagine having a slanted long wall-like object which has its pivot in the center. So you look along the wall. Further aways there’s an object right before the wall. So you should render the wall first and that object last. However just looking at the pivot makes the wall much closer to the camera than that other object, so they will draw in the wrong order. When objects are intersecting it gets even more complex, as mentioned above since you can not really determine which object is before which since intersecting objects technically are both at the same time. At least a part is closer while another part is further away. There is no patent solution.
Doing rendering on a per triangle basis is simply impossible for a game engine as you would need to draw each triangle on its own. So you loose a lot of the benefits of having a pipeline GPU. Also the sorting is really expensive. Also to solve the cyclic issues you would need to dynamically split geometry to solve this properly.
So the best thing you can do is put constraints on the objects placements and different object configurations to ensure you can determine the sorting order. If you use a scriptable rendering pipeline it may make sense to create a modified one that takes care of the sorting based on some of your own metadata. Though this is not really trivial to get good performance.