UI: Have child object(s) automatically scale so they fit into a parent (Scroll rect?)

This seems like it should be something fairly strait forward, but I can only seem to find parents scaling to children on the other way around.

I have a screen in my game, inside that screen I want whatever is there to always be visible, and it’s size can change.
One solution is a scroll rect, where I can have scroll bars, but instead I want the objects inside to scale (zoom in/out) until they are all visible inside the scroll view. (Ideally I also want to be able to manually scroll in/out with the wheel and have the scroll bars appear when zoomed in.)

Is there any way to do this with the scroll rect and content size fit? I have not been able to find or manage it myself; or will I have to script this.

If I have to script it, any suggestions or things to avoid?

Thanks.

Attempting to solve this myself, I have had a few discoveries, but still no method to achieve the bellow example.

  1. Scaling all children can be done by making them all use the stretch Anchor presets and setting the anchor points to the edges of the object for ALL CHILDREN - I am using Better UI to do this.
  2. Fitting to the previous screen can be done with Aspect Ratio Fitter, this will fit the game object with this to fit into the parent object and preserve aspect ratio.

The problem: I am missing a way of re-sizing the parent game object with the Aspect Ratio Fitter when the child objects change size to fit them all.
Utilizing a layout group requires another parent for the children, because it otherwise will re-arrange the child objects. So using the Content Size Fitter is unfeasible.

Please let me know if someone has an example or a tutorial of exactly what is shown below.