I tried having an intermediate GameObject parenting the camera, this is actually my initial setting (in earlier posts, I work on a sandbox project for testing). This is something I would advise generally as it simplifies calculations for a follow camera with a focal point slightly off the target. However I do not see why it would solve the issue of having a smooth camera following a target but both of them updated at different rates (see for instance here for an explanation of what I mean). I believe inserting a parent GameObject between both does not change this fact, regardless of the rate the parent is updated.
About the simple moving average code, you can change the buffer size using simpleMovingAverageRange in the code above. This is the variable you called X. The code does exactly what you described with “true average” but with a lesser computational cost. See for instance the wikipedia page of Moving Average for the maths.