That’s too much. You can do this for tens of objects, not hundreds, let alone thousands.
First, do as you said, infrequent update (not in every frame) of the list.
But before adding them to the list, cull the objects based on some cheap checks: you can use AABB tests (axis-aligned bounding box) which simply test whether the object’s pivot lies between this and that on all three axes.
Regardless doing this for thousands of objects every time introduces a bottleneck of just cycling through that many objects, querying their transforms and doing comparisons. You need a more sophisticated algorithm to do this properly, based on group culling (logical occlusion), where you essentially dynamically build a list of objects that should participate in the final list of objects that are verified.
Once you have your final list (hopefully not greater than 50-100 items), you can try using my distance-checking utility found here . It works with lists or arrays of Vectors, but you can easily customize it.