Using OnDrag() I'm experiencing input lag on touch devices, but not when interacting with mouse controls

Here is a video I took from desktop where there is no input lag. The script functions fine and I get very snappy movement from the OnDrag() using mouse as input.
When I run this on an android build, or play the web build on a touch device, the input lag happens any time I drag the input, the visible circle showing the input position lags behind a fraction of a second and is “smoothed” as if it’s a lerp function, a quick finger movement will cause the white input circle to take a fraction of a secont to smootly glide over to the current position.
I’m wondering if this could be a setting within the phone that determines how it updates current drag position, or if this is a setting we have control over in Unity.

Important note: When the press occurs, the input lag disappears and the input system immediately registers the correct location. If you tap somewhere, let go, then tap far away on the joystick, there is no gliding lerp smoothing and the controls feels super snappy, this lag only occurs when holding the finger down on the joystick.

Here’s a small section of the code I’m using:

public virtual void OnPointerDown(PointerEventData eventData)
{
    OnDrag(eventData);
}

public void OnDrag(PointerEventData eventData)
{
    cam = null;
    if (canvas.renderMode == RenderMode.ScreenSpaceCamera) cam = canvas.worldCamera;

    Vector2 position = RectTransformUtility.WorldToScreenPoint(cam, background.position);
    Vector2 radius = background.sizeDelta / 2;
    input = (eventData.position - position) / (radius * canvas.scaleFactor);
    FormatInput();
    HandleInput(input.magnitude, input.normalized, radius, cam);
    handle.anchoredPosition = input * radius * handleRange;
}