Implementing shadows in a pre-rendered scene?

Graphics Shaders & Materials
1

Hi all,

so a little bit of info on what we’re doing, basically we’re working on a classical survival horror game with static camera angle and pre-rendered backgrounds (think RE 0/1/2/3/REmake) - here’s our blenderartist thread

So i looked for some references on how to setup a pre-rendered scene in Unity, and found this nice tutorial
Basically there’s two cameras, one that renders all the 3d interactive objects in the scene (the player, zombies etc) and another for the background, the background material uses a custom shader that does some stuff with a depth map. Here’s a video to demonstrate

Since I’m no shader/graphics guru, I have zero clue on how to implement shadows; both shadows being cast from the player to the environment, and vice versa.

I’ve been lurking around reading here and there, there are a few terms that I think some of them “could” be related to the problem: Shadow maps, alpha maps, depth maps, shaders, replacement shaders, depth buffer, projection matrix, camera render texture.

I don’t know if my idea is applicable or not, but I thought well the shadows are already there being cast into the collision geometry but since the geometry is not rendered the shadows are not there, so I thought is it possible to map those shadows to the background? … or maybe write a custom shader attached to the collision geometry which makes it completely transparent but still receive shadows…

I’m not sure what the right technique is, so any point to the right direction is greatly appreciated. What do I need to do? what techniques/methods/technology I need to learn to be able to pull this off? (I’m willing to learn anything - currently learning shader programming)

Note:
1- I’m using Unity pro and our target platform is standalone
2- RE 1/2/3 used a blob projector for shadows, while REmake and RE 0 had beautiful lighting and what seemed to be real-time shadows, no idea how they did it though, but at least we know it’s doable…

Thanks!

2

When you say “pre-rendered” backgrounds, you mean they’re just going to be imported into Unity as simple flat images? That’s going to make it very difficult to cast any convincing shadows onto them, as you have no reference as to the depth, geometry, or surface properties of anything in the image… they’re just coloured pixels.

3

@tanoshimi thanks for your response. Yes they’re images (rendered in blender). But we do have depth maps generated for each image. (just like 1:05 in the video) - does that help? (of course along with the level mesh for collision)

4

So I came across this guy yesterday, claims to have gotten lighting and shadows to work properly, not commenting too much except saying “The albedo, i.e. the unlit textures of the scene is used when applying dynamic lighting”. I’m not sure what he’s referring to with the unlit textures of the scene…

5

Bump

6

You’re in for a lot of position reconstruction, so the depth textures are really the key element here.

The shader you will probably spend most of your time in is the background shader. You need to teach it how to both cast and receive shadows and one of the ways to make it happen is through position reconstruction. The idea goes like this: From the position of the image and the depth sampled from the texture, you reconstruct the 3D world position of each pixel. Once you get that right, use that position to sample from the shadow map. Unity provides nice helper functions in the shader to do this.

Then you need to convert that world position to screen space position… that means multiplying it with the view projection matrix, UNITY_MATRIX_VP, just like you multiply object space position in the vertex shader with the model view projection matrix. All you want from that is just the clip space depth, though, which should be the .z component divided by the .w component. For the part about dynamic objects hiding behind the prerendered geometry to work, you also have to output depth from the pixel shader along with the color. Now, this is usually quite slow, because it destroys hardware hierarchical depth culling, but worry not - since it’s a background image, you don’t need to have depth test on. So, unless it breaks something, set ZTest to Always and change the shader queue to Background. That way it renders first and doesn’t overdraw your dynamic objects. Unity got ya covered in this case :slight_smile:

The next step is casting shadows. For that you’ll need to write your own shadow casting pass. Luckily, you can reuse most of the code from the previous part. It’ll read the depth from the texture, reconstruct the world position, convert it to screen space position and output the depth both into the depth channel, but also in an encoded form (should also be in some unity’s helper function) as a standard color output. I suggest reading up on how shadow maps work and it might also be a good idea to look into the generated code of some surface shader and see what the shadow casting pass does.

All this might have a bit of an unwanted effect though with no easy way to prevent it. Your background is going to cast shadows on itself. It should look fine if you render your images without shadows, but if it poses a problem, we’ll have to think of something else. If you’re feeling adventurous, you could also either reconstruct normals from depth or read them directly from another texture, if you have a way to render them easily. Once you have those, you can do the whole lighting of the scene at runtime and have support for dynamic lights and even time of day. Sweet ey?

Either way, best of luck with all that :slight_smile:

1 Like