Reveal alternate texture in shadow

Unity Engine
1

I’m looking for advice on an effect I have in mind. The idea is that I would have 2 textures, one which is visible on lit areas of a model, and one which is visible on shaded areas of a model.

Like so:

1st box would be the base texture
2nd box is the shade texture
3rd box is the 2 combined

My first Idea is that I could have 2 cameras, and 2 objects, and then somehow create a mask from the shadowed areas of the 1st object, then the 2nd object could be overlaid with the mask. This doesn’t seem like a good idea, as I’d have double the geometry in every scene(if this technique is even possible). Any other ways?

I don’t yet know what’s possible with shaders, as I’ve just started working with them, so if someone has some advice on how to do this it could help me figure out where to start on this effect.

2

I think this could be done by rewriting the script on the wiki that desaturates shadowed areas:

http://www.unifycommunity.com/wiki/index.php?title=DesaturatedDarks2

The only problem with that would be that you could only do this for graphics cards that support vertex and fragment shaders.
If that is ok, I think I can help you do that.

Something that might work on older graphics cards would be close to what you suggested. Only instead of having two copies of each object, you would have two cameras. One that somehow rendered the lighting to the alpha channel of a rendertexture, and then the main camera, which then use a simpler shader to blend between the two textures, using the alpha channel from the rendertexture as “selector”.
Of course you need to have Unity Pro for rendertextures.

3

I was looking through the built in shaders to try and understand their behavior a little better, and I noticed that “Lightmapped Diffuse” sort of seems to do what I’m thinking.

here is the result:

and the lightmap:

As you can see, the (quickly made)texture set to be the light map is shown in the shadow, and hidden in the light. Even so, I think some tweaks could be made to enhance the effect.

It would be really nice to be able to mix this with a toon shader, and have be able to set different textures on different “steps” of the toon style shadow.

Forgive the quick drawing:

So thank you for the ideas cblarsen. Do you think this could be possible with a toon style applied?

4

You can already edit the toon shader textures and draw pattens in them.

If you use one of the “lit” toon shaders, the toon cubemap texture will rotate with the direction to the light. It is not the same as revealing different textures depending on the lighting, it is more … well, rotation.

And I don’t think it works well if you have any flat surfaces, because all the pixels on a flat surface have more or less the same normal, and therefore point to the same place in the toon shader cubemap. Therefore no pattern will be visible until the surface curves.

If you want to add this effect on top of toon-shading, you will either have to rewrite the toon shaders, or do the patterned shading as a post-processing step (the method where you draw into two cameras)

I am not promising to write any shaders for you yet. But if the current toonshaders are not good enough, I think a new shader will be needed.

5

This post has been edited countless times. It contains the newest version of this effect. The explanation for how it works is somewhere in the posts below. The package file contains both effects.

Webplayer:Unity Web Player - ShadowRangesLocal

758×352 69.6 KB

Old version below, which is the same kind of shader, but as a full-screen image effect.

I thought, “what the hell, interesting problem”

So I made something, which I think is at least a good start. Now you can look at this, and say if it is not exactly it.

Webplayer:Unity Web Player - ShadowRanges

663×369 102 KB

I included the package with a demo scene.
The smart stuff is in two new image effects placed on the camera

The first image effect simply takes the brightness of the screen, and puts that into the alpha channel

The second image effect uses the alpha channel as a key to select different textures.
You set up an array, where each entry contains

  • The start and the end of the range
  • A pattern texture to use for this range
  • the scaling of the pattern texture
  • A tint colour for the texture. This colour can be partly transparent to allow the original image to show through

Up to two ranges are allowed to overlap at a time so for example it is ok if range 1 overlaps with range 2, range 2 overlaps with range 3, etc.
If two ranges overlap, they will interpolate smoothly between the start of the second range and the end of the first range
If a range does not overlap with others, a bit of performance is saved (2 texture lookups per range instead of 3)

Start with the darkest texture first, and then add the lighter textures in the following ranges.

My textures are pretty crappy, and don’t even wrap properly, but they should demonstrate the concept.

Edit to v. 0.02; package updated with transparency effect
Edit to v. 0.03: tint colours and blending between ranges

73982–2808–$shadowranges_742.unitypackage (363 KB)

6

Wow! That’s a really great effect in the webplayer. I can’t use the package yet as I don’t have pro (hopefully this weekend!). could you publish one with the 2nd effect active?

As far as the one in the webplayer goes, I think it’s really amazing, and I think it could be very usable. I had initially thought to use 2 or 3 textures for each object in order to draw a pattern that could conform to the shape, so I could get a sketchy feel to the shadows. The sort of effect you demonstrate here also has alot of potential for a similar look, but without conforming as much to the shape. Overall, your solution here might offer a really nice look without so much manual labor setting up the shadow sketches for each object. Is it possible to have alpha on the shadow texture, and show a diffuse underneath?

This drawing might Illustrate what I’m thinking of a bit better:

The blue would be the diffuse texture here applied like a standard Diffuse on a UV mapped object. The black sketchy shadow lines would be patterns like you set up. You can see past the sketchy lines, and there is a bit of a shadow still applied in a toon style.

By the way, I really can’t thank you enough for just doing this out of the blue, not only is it a very inspiring effect, but it seems to run very smoothly. You’ll have to let me return the favor in some way! I’ll send you a PM about it.

7

Huh? the 2nd effect is active. The result you are seeing is the combined effect of the two image effects.

And yes, the idea was to do something global, so you don’t have to modify each and every object (and in some cases each and every shader )

Actually I would have trouble figuring out how to do this for an individual object without modifying the objects own shader, since I am having trouble with grabbing the screen buffer in anything but the simplest shaders.

Check the webplayer again. I have added a transparency setting for each range, so whatever was drawn below is allowed to show through.

I’ll update the package above in a sec too.

8

That is truly impressive Carsten. I never cease to be amazed at the talent of the community here!

9

Latest change. The webplayer, package and description above have been updated.

I now allow up to two ranges to overlap, so that they can be interpolated sommothly. This seemed to be necessary to prevent flickering on flat surfaces as the object turns.

Also I extended the transparency parameter to become a full tint colour for each range, since it didn’t really cust much extra.

I don’t think I add much more to it, unless you have any ideas.

10

This is a very compelling effect; especially at first glance.

I say ‘first glance’ because I find if I don’t look too closely or as a screenshot, it’s awesome, but then when do look at it, I see the background plane / effect tile that exists in the shadow of all objects and it detracts from the effect noticeably.

Was it your intention for it to exist in “screen space”? If it were more a local effect, i think it would be more drool-catching (and hold up better to examination).

Great work in any case!

11

I agree. It was not my intention to do it in screen-space. But I haven’t been able to get the GrabPass command to work for me combined with Cg programs. I haven’t gotten any replies to this thread about the subject: http://forum.unity3d.com/viewtopic.php?t=11014

Alternatives would be:

  • Using GrabPass with fixed function combiners. In this case I haven’t found out how to test alpha in more than one direction at once without overwriting the alpha channel.
  • Rewriting individual shaders to finish with this sort of texturing. Actually, this doesn’t sound so bad, now that I am writing it. Because you would probably not need that many different shaders of this type after all. Edit: but you wouldn’t be able to do the texturing based on the combination of the lighting from more than one lightsource. The texturing would be added several times, once for every pixel light.
    I am considering a third method: At some point I saw a depth-to-brightness renderer somewhere. Something like that might be put into a rendertexture and used for a global effect. The depth would then be used to perturb the uv coordinates of the texture.

Thank you.

12

[offTopic]
I’d like to see a “depth-to-brightness renderer”. Would it be camera-centric or could something like that be object-centric as well? I wonder if I could get one to vary the brightness of the surface of object b depending on how near it was to camera/object a. [/offTopic]

Very nice work here!

13

PACKAGE WAS BROKEN

To everybody who tried to download the package, and found it lacking, I found out that I made some error when uploading, so you got an older version. Feel free to try again.

I will be making a completely different take on this though, since Aras helped me get over my fear of _GrabTexture

If you can do any Cg programming at all, it is really quite simple. The depth of a pixel is available as the “fog” part of the V2F_POS_FOG macro.

Pass {
CGPROGRAM
#pragma vertex vert
#pragma fragment frag

#include "UnityCG.cginc"
struct v2f {
	V2F_POS_FOG;
}; 

v2f vert(appdata v)
{
	v2f o;
	PositionFog( v.vertex, o.pos, o.fog );
        return o;
}

half4 frag( v2f i ) : COLOR
{
    float brightness = 1.0 - i.fog;
    return half4( 1,1,1,1 ) * brightness;
}
ENDCG
}

I just typed it into this post, I haven’t tested it, but you “might” be able to paste this into the “Pass” part of a mostly empty shader. I already have a pipeline of 2-3 shaders that I want to write, so I am not going to try too hard with this one right now.

Thank you.

14

Ok, finally managed to make the effect object-centric, instead of screen-centric.

The actual shader is shown on the big “transparent” ball in the new webplayer version.

The red, blue, and yellow ball and the ground use diffuse and toon shaders that have only been extended with extra parameters for three shadow patterns. The actual shaders have not been changed though.

An extra camera draws all the shaded objects once more on top of what the main camera did. However in this camera all the shaders are changed to the transparent shadowrange shader. This gives the same effect as if the shader had been a later pass of the different diffuse and toon-shaders.

A script on each object helps switch the shaders, and all the shaded objects also have to be on a special layer, so the second camera knows which ones to draw.

You might ask: Why didn’t I just make the effect in some extra passes in a normal shader?
I tried. But because the “GrabPass” command seems to only grab the screen once per camera, only one shaded object could be drawn in the correct order.
I tried to postpone the grabpass, but somehow I usually managed to get the grabpass executed before some of the pixel light passes of some other objects.

15

Hello cblarsen,

I used your great work but I have a problem since I used the grass project here.
Object becomes transparent and grass becomes masks.

So I deleted all the stuff about the grass but the biggest problem is here : I don’t know why, but that continues ! :shock:

Here are 2 images :

Note : it’s just with the “Shadow Range Local” I’ve got the problem not with the “Image Effect”.

Could anyone help me please ?
Thank you.

16

Hi there !

Carsten helped kindly me. He allowed me to post his solution.

Here are explanations :

That works fine !
Thank you Carsten :slight_smile:

278287–9977–$shadowrangerevisited_508.zip (349 KB)

17

Hi! This looks promising and I tried your shaders - unfortunately there’s a warning and a error:

Warning: Shader ‘FX/ShadowRange/Toon/Lighted Outline’: no subshaders can run on this graphics card

Error: Assets/Shadow Ranges Image Effect/AlphaRangeToTexture.cs(65,46): error CS1501: No overload for method DrawQuad' takes 1’ arguments

My graphics card is a ATI Radeon HD 4850. And yes, the demo doesn’t work.

18

May be a broken link to a script ?

Have you got a screenshot for each camera’s inspector ?

19

Yep, good hint. All of Scripts were missing (in each camera and object) but the files are there - where do I put which script?