I think I have a half-decent idea how to do it, although I’m still working on the implementation details. We can take advantage of the fact that our sprites are just flat textures. I’m considering using a geometry shader to duplicate the mesh. The first mesh will be billboarded towards the camera (just as it visibly appears on screen.) The second mesh will be billboarded toward the light. Then, in the fragment shader:
- When drawing the light-billboarded mesh, discard all pixels that overlap the camera-billboard mesh.
- When drawing the camera-billboarded mesh, discard all pixels that don’t overlap the light-billboarded mesh. Apply a small depth/normal bias to this mesh away from the view so the shadow won’t land on the visible sprite.
The result of this I believe should be that there will be a cutout in our light-billboarded shadow, in the shape of our visible mesh. That empty space will be filled in by the visible mesh, pushed a short distance away from the view.
There could be a pixel-wide outline in the shadow where the two meshes don’t quite fit together, but we could probably fix that somehow, that is if soft shadows don’t do it for us.
I wish I could use the Stencil buffer for this, but afaik we can only have one shadow caster pass, and from my testing at least I don’t think the Forward Pass is used when writing to the shadowmap (Still fairly new to shaders, so I may be wrong about some of this). If we can use two or more passes I think the stencil buffer could do this pretty easily.
Otherwise I could write some values to the vertexes before leaving the geometry shader and read them in the fragment shader, but I’m not quite sure how I could determine which pixels overlap between the two meshes when I only have access to one fragment at a time.
Still thinking on it.
Edit: I’m pretty sure I have a good way to do this with the geometry shader. I’ll make/post the code tonight. Hopefully I’m not overlooking anythimg major.
Edit 2: Well that’s what I get for hoping. Ran into a few issues. I’m terrible at matrix math and logic-ing my way around transformations, so got pretty hung up on a few things. In particular I wanted to attach an additional set of UVs to the Fragment Input that when interpolated in the fragment shader, would give me the corresponding UV of the other mesh that I could test against. But that requires I determine the mapping in view space between the two triangles, which turns out to be harder than I thought it would be. Unless I’m missing some handy function that can simplify the mapping.
I also noticed a peculiarity where the billboarding on the shadow from the main directional light breaks when the camera gets too close. Spent too much time on it tonight and burnt out a bit, so I’ll probably give it another shot in a couple days.
Edit 3: Alright well, this has been a great learning experience, but I’m going to step away from this idea for now. It’s not that I don’t think this solution can work, but I think it’s a very complicated solution and I can potentially come back to this later when I have more experience. For now, I think it comes down to:
- Either give up on light-billboarded shadows
- Stop receiving shadow in the forward pass and just fake it (maybe using lightprobes?)
- Come up with a better way to do my cutout idea than trying to duplicate the mesh and mapping the screenspace shadow onto the rotated mesh (too much artifacting that I don’t think can be fixed.)
- Or just do what bgolus said was possible and duplicate lights with different lightmasks and have billboarded characters receive light on their own mask (literally duplicating lights in scene, so bad, but technically perfect effect.)
Actually I think I’m going to try the command buffer idea bgolus gave in this post:
Seems like the most correct solution, and might as well learn sometime.