Dragged object puts force on other objects??

In my 2d game I have an object called a bat that I click on and drag around my scene in the game. This bat is supposed to push other objects up once they fall down but the bat can only go so far up. My problem is that when I hold the bat the farthest up it can go, against the ceiling, whenever the objects drops on it they kind of bounce back up as if they got hit up again.
I figure that this is because I am adding velocity to the object to drag it around. At first I simply added mass to the bat but this only works in the editor, as soon as I build it and run it on a phone it starts to do it again.

Here is how I move the bat around:

    void OnMouseDown()
    {
        if (!firstTouch)
        {
            timer.GetComponent<TimerController>().startTimer = true;
            firstTouch = true;
        }
        Vector3 worldPos = Camera.main.ScreenToWorldPoint(Input.mousePosition);
        worldPos.z = transform.position.z;
        offset = worldPos - transform.position;
    }

    void OnMouseDrag()
    {
        Vector3 curScreenPoint = Input.mousePosition;
        Vector3 curPosition = Camera.main.ScreenToWorldPoint(curScreenPoint);
        curPosition = (curPosition - (transform.position + offset)) * 20;
        rb2d.velocity = curPosition;
        rb2d.velocity = Vector2.ClampMagnitude(rb2d.velocity, 20);
    }

Can anyone help me out?

I would use a TargetJoint2D attached to the ‘bat’. You can enable/disable it as appropriate.