There are a few main ways to go about drawing outlines / edges on objects.
Fresnel - Use a clamped dot(view normal, view direction).
Pros: Works in shader without a separate pass. Will highlight almost every edge on smooth shapes. Cheap. Works on transparent objects.
Cons: Difficult to control line thickness as its dependent on the angle of the surface to the camera rather than the depth and the roundness of the surface. Impossible to have consistent line thickness. Doesn’t work well on hard edged shapes or low poly meshes. Lines are only drawn on the “inside” edge.
Push Hull / Mesh Doubling - Duplicate the mesh and expand along surface or view normals, full front faces. Unity’s toon outline shader works this way.
Pros: Cheap on low poly characters. Works on pretty much every device. Gives a lot of control for line thickness. Lines are outside the edge.
Cons: Lines still won’t be constant, and only works on smooth meshes. Hard edges will create holes in the outline. Thicker outlines can make smaller details and concave edges loose their outline as its dependent on the outline mesh not intersecting other geometry.
Edge Detection (Depth based) - Test depth between multiple screen pixels.
Pros: Draws edges on everything that’s opaque. Precise pixel size control over line width.
Cons: More expensive than other options, but constant cost regardless of geometric complexity. Will “edge” surfaces that are simply angled away from the camera because the depth discontinuity between pixels is high enough on glancing angles without heavy depth biasing and then will miss seemingly obvious edges because of the bias. Each pixel of thickness increases the cost exponentially. Doesn’t
Edge Detection (Depth & Normal based) - Test depth and normal between screen pixels.
Pros: Everything above, plus catches edges depth only solutions don’t.
Cons: Even more expensive.
Geometry Shader Edge Detection - Test adjacent polygons to find corners, construct outline geometry.
Pros: Catches every single edge, guaranteed. Complete control over outline thickness. As close to hand drawn edging as you can get.
Cons: Super, super expensive, especially on high poly objects. Also, doesn’t work at all in Unity since they don’t provide adjacency information to the geometry shader.
Geometry shader based outlines are kind of the holy grail of outlines, but as I said they’re stupidly expensive (a 100,000 polygon model can bring a modern computer to it’s knees), and they don’t work in Unity.
So, lets go back to Edge Detection. The depth & normals (Roberts Cross as it’s called in Unity’s Image Effect) method is really quite good, but requires you’re using the deferred rendering path, or your camera has to render depths and normals for all opaque scene geometry. Obviously that’s not working for you for some reason.
Unity’s Sobel implementation misses a ton of edges because of how it handles the issues of depth discontinuity between pixels on glancing angle geometry. The depth disparity needs to be greater and greater the further away something is so edges disappear on small details as they get further away. This works for the style a lot of people are going for when doing outlines. You can write your own by not increasing the disparity required in the distance, but you might start running into issues of depth precision and that glancing angle issue I mentioned before. However, unless you need really thick outlines, this really is the best you can get.
If you don’t want to use an image effect, you can do it as an extra pass in the shader. This can let you limit outlines to a specific object instead of applying it to an entire scene. This is actually something I’ve been working on lately. You can only do the depth based edge detection when doing this, unless you’re using deferred, but it offers a bit more control than the other options.
