alpha blend blurry highlight outline(PS's outer glow) with different color per outline, possible?

[colin299]

hello, I am writing some shader code to provide a blurry outline to target object in a 3D game.
so far so good, it works with depth also because I reused the depth buffer of original scene render.

But once 2 target objects need the outline & they have different outline color & they are close together, the outline color will be wrong(purple part)

to explain this behavior, I need to show all the rendering steps
1.first I render all target objects using replacement shader to a separate RenderTexture, just like the following 2 images
(here I reuse the original scene depthBuffer for ZTest, but do not write to it)

rgb part

alpha part

(continue in reply, unity only allow 5 images upload)

[colin299]

then I “bleed” the rgb part of this RenderTexture by pingpong between 2 RenderTextures at 1/4*1/4 resolution



using the following method

result.rgb = max(max(max(colA.rgb, colB.rgb), colC.rgb), colD.rgb);

it just sample color of 4 corners next to a pixel, and output the maximum
becuase of this “wrong” method to “bleed/expand” pixels, the purple part is produced(which I don’t want)

[colin299]

at the same pingpong passes, it also “blur” the alpha channel

it just average a pixel’s 4 near corner’s alpha value(like kawase blur filter, but with constant offset)

result.a = dot(fixed4(colA.a, colB.a, colC.a, colD.a), fixed4(_Intensity, _Intensity, _Intensity, _Intensity));

at last,
I subtract the blurry alpha by the original alpha to get the correct alpha value for final composite pass,

so in the final composite pass, I will have a correct alpha value to use for alpha blending.

[colin299]

currently the alpha channel is correct, but the rgb part is not(see the purple part, I don’t want it)

So why I need to “bleed/expand” only the rgb part & at the same time do a regular blur only to the alpha channel? it is because I just can’t regularly blur the rgb part, as it will mix black pixel colors to the outline color, just like this

although the rgb value between blue part & red part is correct(correctly mixing red & blue), the outline itself is mixing unwanted black color from the black background when blurring, it makes the outline very dirty(it is an expected result if you regularly blur the rgb part with a black background).

[colin299]

the question is, how to get correct blurry outline rgb when object are close together/overlapping?
I believe the only problem now is “how to correctly bleed/expand pixels in fragment shader”

so input is:

currently I can only do this(but it is wrong if 2 target objects overlap)

what I want is this(I photo shop this result)
with this correct rgb value & the “already correct” alpha value, I can do correct alpha blending in final composite pass

is it possible in Opengles2? I need to make it fast enough to run in mobile…
any help/suggestion are welcome!

if there are better(faster) method to do blurry outline, I really want to know also.

[bgolus]

Try using the blur method you’re using in post #4, and do a premultiplied blend (Blend One OneMinusSrcAlpha) rather than straight alpha blend (Blend SrcAlpha OneMinusSrcAlpha) when compositing the glow back into the scene. It might “just work” with that.

[colin299]

I tested your idea, you are right.
“blurring rgb with black background” is just really similar to “bleed rgb and then multiply blurry alpha”.

the changes I made accroding to your idea:
(1)changed the pingpong pass from “bleed rgb & blur a” → “just blur rgba together”
(2)changed the final composite pass from “straight alpha blend (Blend SrcAlpha OneMinusSrcAlpha)”
to “premultiplied blend (Blend One OneMinusSrcAlpha)”

the result is good.

but there is a new problem, It only works if I write alpha as 1 when doing replacement shader rendering

after the edit, reduce the alpha value will not fade out the outline, but instead, make it glow more.

---

red outline object write(1,0,0,1) to outline RenderTexture when doing replacement shading
the result is correct.

but when I write (1,0,0,0.5), in the past, I could make the outline fadeout 50%.
now it just glow more.

when set alpha to 0, in the past, outline will disappear completely.
but after the edit, it just stays and glow more

this is the method I use in final composite pass,

//use when blur rgba in pingpong, it means bloom's rgb was pre-multiplied by alpha
//col = scene original color
//bloom = the RenderTexture of pingpong's result

col.rgb = col.rgb * (1-bloom.a) + bloom.rgb;

is it an expected behavior or maybe I did something wrong?

The reason why setting the outline alpha to zero will not fadeout the outline but make it glow more, is because when alpha is 0, final composite pass output “original scene color + whatever in the outline rendertexture’s rgb”.(so it make sense why setting outline alpha to 0,make it glow more.)

[a little fix after the above edit accroding to your idea]
Maybe I should pre multiply the alpha to the rgb before the pingpong blur start?
-I did this, I can fadeout outline now, it is 100% correct when alpha is 0 or 1, but setting alpha between 0~1 is not exactly correct(it seems the color is darken when alpha is 0.5, currently no idea how to improve this)

[bgolus]

I mean, that is expected behavior for premultiplied alpha. If you just reduce the alpha it changes from alpha blend to additive. Applying the alpha to the color at the beginning is usually the way to resolve this, but I suspect that’s causing problems with your blur technique. I don’t really have a solution for you on this, though you might be able to hack around it by applying a gamma curve or power to the alpha before applying it to the color pre-bloom.

Try something simple like color * pow(alpha, 2.0) pre-bloom and see if you get results that are more appealing.