EventSystem detecting a recttransform that is not interactable!

Unity Engine
1

Hovering the mouse over the first screenshot is fine, it sends back the correct index and data.
However, I only want the background of the sprite to be interactable and not the recttransform in it’s children.
When the player holds their mouse they can drag this rect transform anywhere (works). However, while dragging this rect transform it’s still detecting the index of the background border!. Is there any way to fix this? Also I can confirm that hovering in and out is detecting correctly, I just need to find a way to make this sprite uninteractable.

7943695--1016398--upload_2022-3-5_17-4-58.png

InspectorData.

7943695--1016410--upload_2022-3-5_17-8-49.png
7943695--1016410--upload_2022-3-5_17-8-49.png475×678 41.9 KB

7943695--1016413--upload_2022-3-5_17-9-22.png
7943695--1016416--upload_2022-3-5_17-9-53.png

2

Also here is the code:

    public void OnDrag(BaseEventData data,int index){
        if(InventoryManager.Instance.inventory[index].item == null){
            dragIndex = -1;
            return;
        }
        dragIndex = index;
        Canvas canvas = UiManager.Instance.canvas;
        PointerEventData pointerData = (PointerEventData)data;
        Vector2 position;
        RectTransformUtility.ScreenPointToLocalPointInRectangle(
        (RectTransform)canvas.transform,pointerData.position,canvas.worldCamera,out position);
        inventoryPanel[index].childRect.position = canvas.transform.TransformPoint(position);
    }
3

Resolved, the TMP_Pro component was raycastable