Encapsulate bounds not correct with UI scale mode

I’m trying to encapsulate recttransforms children with a single box which works fine with a canvas with “Constant pixel size” but once I change it to “Scale with Screen Size” they are not encapsulated correctly.
Does anyone have any idea what’s wrong?

Constant pixel size example
5986298--643502--bounds constant pixel size.png

Scale with screen size
5986298--643499--bounding scale with screen.png

The code

public class GetBounds: MonoBehaviour
{
    public List<RectTransform> rects;

    private void Update() {
        if (Input.GetKeyDown(KeyCode.D)) {
            CalculateBounds(GetComponent<RectTransform>(), 1);
        }
    }

    void CalculateBounds(RectTransform transform, float uiScaleFactor) {
        Bounds bounds = new Bounds(transform.position, new Vector3(transform.rect.width, transform.rect.height, 0.0f) * uiScaleFactor);

        foreach (RectTransform child in rects) {
            Bounds childBounds = new Bounds(child.position, new Vector3(child.sizeDelta.x, child.sizeDelta.y, 0.0f) * uiScaleFactor);
            bounds.Encapsulate(childBounds);
        }

        transform.sizeDelta = bounds.size;
        transform.position = bounds.center;
    }
}

Best way is to set the size delta to zero and instead set the anchor min / max.