Character Controller follow mouse

I searched for a while and found nothing that worked. I am trying to get my character controller to follow the mouse when the right mouse button is clicked. I got it working using trasnsform.position and Vector3.lerp, but am having some issue converting it to the character controller. I have tried a number of things, but none seem to work.

Vector3 targetPoint = ray.GetPoint(hitdist);
if(Input.GetMouseButton(1)){
				//transform.position = Vector3.Lerp (transform.position, targetPoint, Time.deltaTime * 5);
				CharacterController controller = GetComponent<CharacterController>();
        		//moveDirection = new Vector3(Input.GetAxis("Horizontal"), 0, Input.GetAxis("Vertical"));
				moveDirection = new Vector3(targetPoint.x,0,0);
        		moveDirection = transform.TransformDirection(moveDirection);
        		//Multiply it by speed.
       		 	moveDirection *= speed;
        		//Applying gravity to the controller
       		 	moveDirection.y -= gravity * Time.deltaTime;
        		//Making the character move
        		controller.Move(moveDirection * Time.deltaTime);
			}

Thanks for your help!

This is a variation of the mouse orbit that I use in my game, just have another script that enables this one when you release the right mousekey

public GameObject target;
	private Quaternion myRotation;
	public float distance;
	public float xSpeed;
	public float ySpeed;
	public int yMinLimit;
	public int yMaxLimit;
	private float x;
	private float y;
	private Vector3 angles;
	

	// Use this for initialization
	void Start () {

			distance = 10.0f;
			
			xSpeed = 250.0f;
			ySpeed = 120.0f;
			
			yMinLimit = -20;
			yMaxLimit = 80;
			
			x = 0.0f;
			y = 0.0f;
		    angles = transform.eulerAngles;
		    x = angles.y;
		    y = angles.x;
				
	
	}
	
	// Update is called once per frame
	void Update () {
		
			xSpeed = 12.5f;
			ySpeed = 6.0f;
		
			if (target) 
			{
				x += Input.GetAxis("Mouse X") * xSpeed * 0.02f;
				y -= Input.GetAxis("Mouse Y") * ySpeed * 0.02f;
				y = ClampAngle(y, yMinLimit, yMaxLimit);
				myRotation = Quaternion.Euler(y, x, 0);				
				transform.rotation = myRotation;
			}
				
}
	float ClampAngle (float angle, float min, float max) 
	{
		if (angle < -360)
		{
			angle += 360;
		}
		if (angle > 360)
		{
			angle -= 360;
		}
		
		return Mathf.Clamp (angle, min, max);
	}

Never mind, solved my own question. I simple need to use

moveDirection = new Vector3(0,0,SomeSpeed);

where SomeSpeed is a float. Thanks for the help though!