Camera constantly jumps between two locations with my simple drag camera script

I’m trying to make a very simple script that will move the camera as I drag the mouse while the right button is down, and while it sort of works as I expect it to the camera keeps flickering between two positions with every update.

using UnityEngine;
using System.Collections;

public class CameraControls : MonoBehaviour
{
	Camera cam;
	Vector3 oldPos;
	Vector3 pos;
	Vector3 deltaPos;

	// Use this for initialization
	void Start ()
	{
		cam = Camera.main;
		Debug.Log ("start");
	}
	
	// Update is called once per frame
	void Update ()
	{
		if (Input.GetMouseButtonDown (1))
			oldPos = cam.ScreenToWorldPoint (Input.mousePosition);

		if (Input.GetMouseButton (1)) {
			pos = cam.ScreenToWorldPoint (Input.mousePosition);
			deltaPos = new Vector3 (oldPos.x - pos.x, oldPos.y - pos.y, 0);
			cam.transform.position += deltaPos;
			Debug.Log (oldPos.x + " " + pos.x + " " + (oldPos.x - pos.x));
			oldPos = pos;
		}
	}
}

Like I said, put oldPos = pos outside.

But you have 2 more problems with your code:

ScreenToWorldPoint depends on the z coordinate you pass to it. See this.

Furthermore in unity the y-Axis is the axis to the sky. So I had to change that.

using UnityEngine;
using System.Collections;

public class CameraDrag : MonoBehaviour {

	Camera cam;
	Vector3 oldPos;
	Vector3 pos;
	Vector3 deltaPos;

	// Use this for initialization
	void Start() {
		cam = Camera.main;
		Debug.Log("start");
	}

	// Update is called once per frame
	void Update() {
		if (Input.GetMouseButtonDown(1)) {
			Vector3 mousePos = Input.mousePosition;
			mousePos.z = 1;

			oldPos = cam.ScreenToWorldPoint(mousePos);
		}

		if (Input.GetMouseButton(1)) {
			Vector3 mousePos = Input.mousePosition;
			mousePos.z = 1;

			pos = cam.ScreenToWorldPoint(mousePos);

			deltaPos = new Vector3(oldPos.x - pos.x, 0, oldPos.z - pos.z);
			cam.transform.position += deltaPos;
		}

		if (Input.GetMouseButtonDown(1)) {
			oldPos = pos;
		}
	}
}

If you want 1 to 1 movement of your camera to your mouse movement, then you would have to know the distance from the camera to the ground. For that you would have to use a Raycast. Or if its constant change mousePos.z = 1 to your distance.