I am trying to replicate HandleUtility.DistanceToLine() because that function only works in an Editor script and I want it to be called from another script. I use WorldToScreenPoint() to get my line in screen space and then test it to against Event.current.mousePosition (y inverted).
It all works kind of ok but I find that HandleUtility.DistanceToLine() is more accurate. Would the Unity team be so kind to disclose the inner magic of that function?
I forgot to set the Z distance to zero. Works fine now. I made these replacement functions:
MouseDistanceToLine() as a replacement for HandleUtility.DistanceToLine()
and
MouseDistanceToCircle() as a replacement for HandleUtility.DistanceToCircle()
It works both in Editor and in Play mode. You can find the functions here:
http://wiki.unity3d.com/index.php/3d_Math_functions
Forum discussion:
http://forum.unity3d.com/threads/126575-3d-Math-functions