ScreenToWorldPoint offset affecting ScreenPointToRay ?

Hi, all. I recently changed a project from an orthographic camera (which was super easy and intuitive) to a perspective camera and I’ve run into some problems with accurate dragging in world space.
I’ve got one script in which I use ScreenPointToRay to follow the mouse around and get collision info from certain objects (which seems to work fine):

void Update () {
			Ray toMouse = Camera.main.ScreenPointToRay(Input.mousePosition);
			RaycastHit rayInfo;
			bool didHit = Physics.Raycast (toMouse, out rayInfo, 500.0f);
			if(didHit) {
				// Do some stuff
				}
	}

And I have a different script which uses ScreenToWorldPoint to drag cards around the screen:

void OnMouseDrag (){
			Vector3 temp = Input.mousePosition;
			temp.z = dragOffset; // I'm using an offset of 8f, which seems to give me a fairly accurate drag on screen
			Vector3 cardPosition = Camera.main.ScreenToWorldPoint (temp);
			transform.position = new Vector3 (cardPosition.x, cardPosition.y, -2f); //z position is adjusted so that the card passes in front of other objects rather than behind or through.
		}

My problem is that as soon as I use that drag script with the offset, my raycast is also offset and I no longer get accurate collision results.
I can’t figure out why the drag offset in the ScreenToWorldPoint script would affect the ScreenPointToRay script. Should they not be used together? Even though they’re in different scripts?
Any pointer in the right direction would be helpful! Thanks in advance!

UPDATE: Alright, I think I’ve solved my own problem.
Instead of using ScreenToWorldPoint in the second dragging script I used another ScreenPointToRay and instead of doing it in the MouseDrag () function I did it in Update (). That seems to have given me much better dragging results. It looks like this now:

public GameObject dragSurface;
public Vector3 dragCoords;
public LayerMask hitLayers;

void Update () {
    		Vector3 mouse = Input.mousePosition;
    		Ray castPoint = Camera.main.ScreenPointToRay (mouse);
    		RaycastHit hit;
 
    		if(dragSurface.GetComponent<BoxCollider>().enabled) {
    		     if (Physics.Raycast(castPoint, out hit, 500.0f, hitLayers)) {
    			dragCoords = new Vector3 (hit.point.x, hit.point.y, -2.0f);
    		     }
    		}
    	}

MouseDrag () now just updates the position of the card using dragCoords.
I still needed the card to be dragged in front of other objects so I created a collider called dragSurface closer to the camera to cast the ray into. This works great to reduce the dragging FOV offset but it caused problems by blocking all of my other MouseDown collisions. No masking or physics layer matrix trickery would ignore it so I hacked together a fix that basically just turns its collider on when I need to drag something and then turns it off again.
Kind of ugly, but oh well.
Maybe a plane or something would work better than a box collider? I dunno. I’d love a more elegant solution.