Offsetting rotation depending on camera angle

Unity Engine
1

I have a character object that has a child object “rayPoint” that is used the hold a fire ray(think D3 disintegrate ray).
When I click on the floor I want the parent object to rotate so that the ray is going through the clicked position.

Like this:
https://www.dropbox.com/s/25jywv61cgws95n/d3 ray.JPG?dl=0

What I get is this:
https://www.dropbox.com/s/pfp2aietchqbltn/ray position.JPG?dl=0

The silver coin is marking the targetPosition. As you can see it is off because the rayPoint is 1.2f higher than the player.
Here is the code for that:

positionToLookAt = new Vector3(targetPosition.x, Position.y, targetPosition.z);
ParentTransform.LookAt(positionToLookAt);

What I would like to do to make up for it is rotate the player more so the ray is now going through the clicked position again.

So I figured out if I subtract the local postion of the rayPoint child object I get close.
https://www.dropbox.com/s/s467ybrlmeludot/ray position2.JPG?dl=0

Here is the code for that:

Vector3 target = targetPosition - rayPoint.transform.localPosition;
positionToLookAt = new Vector3(target.x, Position.y, target.z);
ParentTransform.LookAt(positionToLookAt);

And that works pretty well as long as the camera stays at a 45 degree angle.

If I move camera the ray is no longer going through the clicked position.
Here is a picture with the camera at a 90 degree angle.
https://www.dropbox.com/s/wat35b4wi01eqnz/ray position3.JPG?dl=0

I figure I must somehow limit how much I subtract from the position depending on the camera angle but I’m not sure how to tackle this problem really.

Any help or insight would be great.

2

So I guess subtracting was just luck that it was close because if I rotate the camera the position is way off.