Camera Panning fail - erratic movement and then stops

my script is supposed to find the difference between two raycast intersections with a plane and move the camera accordingly. On the beginning of a touch a ray is cast from screenpoint and then on the move phase another ray is cast. This repeats every frame while the finger is still on-screen and moving. I essentially want to calculate the delta position of each raycast and then move the camera intuitively in a “panning motion” (my reasoning for using rays and not just the touch positions is so that i can acheive a more realistic panning effect where it would appear that the finger is actually dragging the landscape itself similar to clash of clans but with a perspective camera instead of an ortho.

public class panTest : MonoBehaviour {

public Plane panPlane;

Vector3 oldPoint;
Vector3 newPoint;
Vector3 deltaPoint;

// Use this for initialization
void Start () {
    panPlane.SetNormalAndPosition(Vector3.up, Vector3.up * 5);
}

// Update is called once per frame
void Update () {
    if (Input.touchCount == 1)
    {
        if (Input.GetTouch(0).phase == TouchPhase.Began)
        {
            Ray panRay = GetComponent<Camera>().ScreenPointToRay(Input.GetTouch(0).position);
            float hitDist;

            if (panPlane.Raycast(panRay, out hitDist))
            {
                Debug.Log("Hit");
                oldPoint = panRay.GetPoint(hitDist);
            }
        }
        if (Input.GetTouch(0).phase == TouchPhase.Moved)
        {
            Ray panRay = GetComponent<Camera>().ScreenPointToRay(Input.GetTouch(0).position);
            float hitDist;

            if (panPlane.Raycast(panRay, out hitDist))
            {
                Debug.Log("Hit");
                //gets difference in finger positions
                newPoint = panRay.GetPoint(hitDist);
                deltaPoint = newPoint - oldPoint;
                oldPoint = newPoint;

                //moves camera according to diff in finger pos'
                transform.Translate(-deltaPoint.x, 0, -deltaPoint.z);
            }
        }
        if (Input.GetTouch(0).phase == TouchPhase.Ended)
        {

        }
    }
}

when i run this script and touch the screen and move my finger the camera moves several million units in a direction and then fails and becomes completely unresponsive. I am very confused as to why this happens so any help would be greatly appreciated

P.S. if anyone has another way that this effect can be achieved with a perspective camera PLEASE let me know!! I am desperate for a working camera panning control (using touch) that is intuitive, smooth, and uses this “touch point- accurate panning” effect

This is the code we use to make the camera pan left and right.

 using UnityEngine;
 public class ScrollCamera : MonoBehaviour
 {
 public float dragSpeed = 1;
 private Vector3 dragOrigin;

 void Update()
 {
 if (Input.GetMouseButtonDown(0))
 {
     dragOrigin = Input.mousePosition;
     return;
 }
 
 if (!Input.GetMouseButton(0)) return;
 Vector3 pos = Camera.main.ScreenToViewportPoint(Input.mousePosition - dragOrigin);
 Vector3 move = new Vector3(pos.x * dragSpeed, 0);
 
 transform.Translate(move, Space.World);  
 var posi = transform.position;
 posi.x =  Mathf.Clamp(transform.position.x, -1.0f, 1.96f);
 transform.position = posi;
         }

    }

Just stick it on your main camera and test it out.