It seems you may have leaped from a performance issue to assuming it’s a data structures problem. It may well be, but prove it first, otherwise you’re wasting time doing work that won’t help.
DO NOT OPTIMIZE CODE JUST BECAUSE… If you don’t have a problem, DO NOT OPTIMIZE!
If you DO have a problem, always start by using the profiler:
Window → Analysis → Profiler
Failure to use the profiler means you’re just guessing, making a mess of your code for no good reason.
https://discussions.unity.com/t/841163/2
Notes on optimizing UnityEngine.UI setups: