Dragging UI elements in Screen Space canvas

I’m trying to drag and drop items on screen space canvas. This is closest I’ve got it working.

It offsets more and more when you get further away from starting point of drag. I think I’m missing some scaling in positioning the UI-element?

I want it to stay right on mouse position on screen when I drag the UI-item. I’m using screen space canvas, because I want to have 3D-elements in the UI too.

This is from UI-element I’m trying to drag:

    public void OnDrag(PointerEventData eventData)
            Vector3 pos = new Vector3(Input.mousePosition.x - Screen.width/2,
                                       Input.mousePosition.y - Screen.height/2, 0);
            transform.localPosition = pos;

This is how it should be done:

public void OnDrag(PointerEventData eventData) {
    var position = WorldCamera.ScreenToWorldPoint(Input.mousePosition);
    transform.position = position;