What should I do to debug this raycast?

Physics.Raycast (mytrans.position, direction, hit, range,layerMask)

I want to do the usual raycast linedrawing but it doesn't function as I want it to.

Use Debug.DrawRay.

var hit = Physics.Raycast (mytrans.position, direction, hit, range, layerMask);
Color color = hit ? Color.green : Color.red;
Debug.DrawRay(mytrans.position, direction, color);

This will cause the ray to become green when it is a hit and red when it is a miss.