I saw this earlier question about vertext lighting, http://forum.otee.dk/viewtopic.php?t=1256&highlight=shadow, but the post seemed to get bogged down in all kinds of other shadow/lighting techniques.
The technique of vertex lighting lets you bake lighting/color information into the vertices of the mesh itself so that you can get the visual effect of lighting without calculating them every frame. It’s similar to baking the lighting into the texture, but it has a massive advantage in that you don’t have to create a new texture for every single part of your set.
This technique works for fixed shadows, so if you’re going to have a dynamic light in the scene, this isn’t the best way to go. But for wherever you have static lights, it’s an extremely efficient way to get a whole bunch of depth and shadow into your game. So it’s typically used with static set pieces, and then different kinds of shadows are used for interacive entities.
However, I tried baking some lighting into the vertices of a Maya object, but apparently Unity is not reading in any of this data. Am I just not enabling vertex lighting somehow or is Unity just not able to implement vertex lighting at this point?
An example of vertex lighting is in the below image where the light appears to be casting a shadow onto a tiled image (just in front of the green guy).
Unity seems to always be ahead of the curve though, so I’m guessing they just have a different way of getting all these static shadows into the set of a game. I just can’t figure it out.
Thanks!
Vertex lighting and baked lighting are different things. Vertex lighting can be turned on per light basis, and only works out the calculation of a light dynamically with one polygon, rather than all the pixels separately, and can do quite a speed boost.
Baked lighting has been done before in maya, and can be seen with this example here: On The Edge - Det bedste fra tech verdenen
It uses up a lot of VRAM though, and isn’t recommended. However, a possible alternative is to multipass another texture, a transparent light map texture or just use a light map mask on a black texture on top of the original texture. For information how to do the masking technique, I’d follow this thread: http://forum.otee.dk/viewtopic.php?t=1626
[EDIT] Is there a way to scale the preview of an image on the forum, like in html?
[EDIT EDIT] It’s just unreadable like that, I changed it to a link.
Ah yes, I should have specified “vertex-baked lighting” as EA calls it because “vertex lighting” alone refers to the real-time lighting you have described. We can’t really call it just “Baked Lighting” though because that could also refer to texture-baked lighting. All this terminology starts to get me dizzy 
But I don’t know why you’re saying that this type of static shadow creation isn’t recommended. It was a standard method back on our PlayStation games as well as on our latest PC game. That’s why this baking is a standard feature in Maya and has been for years. I can’t quite imagine where another solution would be easier or less resource-intensive for filling an entire set with static shadows.
The latest dynamic lighting setups are niftier, but they really need to be blended with this established method to build a game that will run blazing fast on older hardware.
More particularly, I would dearly like to know how you got your vertex-baked lighting effect into that screenshot. It’s very nicely done. I just hope the answer isn’t that you have to use thousands and thousands of polys.
Maybe I just took for granted how good our other engine is at easily filling entire scenes with great-looking static shadows.
Ah, there was a misunderstanding. Firstly, it isn’t my screen shot. Secondly, I thought you meant texture baking. Vertex light baking isn’t supported afaik, and vertex lights are very quick - unity doesn’t support shadows at the moment so dynamic stuff is as fast as the static stuff over on the playstation, and co, if not a lot faster.
Your static shadowing options were laid out by me in the last post, as well as that other thread following shadows that you referenced in the original post. Projecting shadows maybe a good option too.
You could try using a self illumination shader, and only illuminate some bits. This makes part of the texture rendered brighter, and could be what you want.
Actually you can do baked vertex lighting.
We do import vertex colors from maya and you can use them from inside shaderlab. At the moment our builtin shaders don’t use the vertex color though so you need to write your own shader that uses the vertex color.
I will look into providing the diffuse shader with baked vertex lighting support tomorrow.
If you could just send in one of your maya files with textures, that uses baked vertex lighting and a screenshot of how it should look, that would help a lot in making sure that the shader looks right.
Use Report Bug.app to send your model.
Joachim,
That’s fantastic! I have to run just now, but I will get them to you tomorrow. Thank you!!
Thank God this is finally getting done!!! And thanks bigbrainz for bringing this issue up again and I guess explaining the need for this shader better than I could before 
Joachim has been waiting for me to get him a sample file from Maya with the vertex lighting effect we need so that he can get his shader working correctly.
But I have to admit that it’s kicking my butt. I’ve been working on it for hours and I can’t get Maya’s vertex-baking to generate anything nearly as good as the effects we’re getting in our old engine [see screenshot].
Notice in this screenshot how you have some nicely contrasting bright patches and dark patches, and you can see how the torches are casting shadows from the door frame onto the door behind it. It’s subtle, but it definitely does a great job of adding depth.
In Maya the shadows come out really blocky. It’s almost impossible to get much contrast between light/dark patches. And the colors all really bleed together. I was trying to use no falloff, linear falloff, even quadratic falloff. Just nothing is working yet. I’ve asked my guru buddy to see if he can help me out.
I’m starting to think that I just really took for granted how smoothly this worked in our old engine, and I’m not really sure what to even ask Unity to do about it yet.
Even if real-time shadows were working, that’s just a HUGE performance hit on the game, so I’m going to do everything I can to get the vertex-baking working, but so far I’m stuck. 
I sent Joe a scene too, and yes it takes a lot of tweaking to look good in Maya… I just sent him a quick and dirty example. But to get good results in Maya you have to use Mental Ray and do multiple passes. One or two occlusion passes and then some light only passes with it set to multiply and turn up the occlusion rays up to a couple hundred.
Yeah, your screenshot has some very nice blending happening on the vertex colors that must be handled by whatever engine you were using at the time. But I’d like OTEE to just at least give us a preliminary shader to play around with, even if it’s not perfect. I did a bake last night on the level I’m working on, and even though it was a little blocky, the lighting just looked so much nicer then without it.
I’ve been asking for this feature for months now and It’s just great to see the ball start rolling on it.
Great tip. I’ll start messing with Mental Ray tomorrow then. If you get any samples of how good it can get, that would be inspirational. And I agree that getting something functional is fantastic so that we can start messing testing and tweaking a bit on our own.
Bigbrainz, I don’t know what version of Maya you have, but Maya 7 has added some great features to the “Paint Vertex Color Tool”. The feature we need is “smooth”… select your mesh after you’ve baked some lights/shadows into it and go into “Paint Vertex”. Turn the color down to zero/black and choose smooth blending, then “flood” the entire mesh a couple of times to smooth out the colors/shadows across the mesh. This should do the trick… I wish I could test this myself, but I’m still on Maya 6.5.
The other trick is of course to tesselate the mesh around the important shadow/light spots to get more detail.
Here is the first version of the baked vertex light shader.
I basically adds the baked vertex lighting on top of all other lights you might have in your scene.
10990–418–$bakedvertexlight-vertexlit_688.shader (1.92 KB)
WooHoo! That is so fun to finally see and play around with! Joe, in case you didn’t know this already, you are the man
It looks almost identical to the bake in Maya, if not actually a little smoother because you’re averaging it against existing lights in the Unity scene.
Very nice first stab at this shader!
Thanks a million!
Actually Peter wrote the shader. But thanks.
School of Rock reference?
Anyone have a screenshot of this new shader in action?
Yeah, a before and after would be cool.
Does no one else use a 3D program that can bake some Vertex Shadows/Colors/Lights/Vertex Pre-Lighting (have to cover all the terms there there’s no dispute what we’re all talking about)?
I would create a good example but the project I’m working on should’ve been done a week ago and I can’t take the time to tweak out a nice screenshot right now. But I will say that the shader works/looks identical to the Vertex Coloring you get in Maya (and most likely any other 3D app that has this feature).
Look at bigbarainz screens there and pretend it looks bland and washed out with no shadows… that’s your before shot. Or look at your favorite videogame and pretend it looks bland and washed out. Baked Vertex Colors are used in every current gen game on the market to build contrast, break up repetition on tiled textures, and remove dynamic lights at run-time to save CPU cycles.
lol, no “School of Rock” reference, it’s a good movie though. Joe has just been very helpful lately and he deserves some thanks. There’s a layered shader coming out soon too that will help with textured lightmapping… a shader that has multiple textures, and more importantly, multiple UV sets. This is great not just for lightmaps, but also for layering multiple textures together for dirt/grime and other anomalous details. To be able to place this stuff precisely with multiple UV sets is an awesome addition to Unity.
Never needed it before…The Quake/Unreal/Torque engines did all that for me. 
(Note this was up to the Quake3/Unreal Tournament era, don’t know what they do now with their fancy schmancy bumpmaps and vertex shaders.
