Make GameObjects dynamically visible/invisible for certain cameras

I have multiple cameras (no static amount, new cameras can be created, but usually it should be less than 20), and have certain GameObjects (no fixed amount, could be 10, could be 40), I’ll just call them “Toggleable GOs” here.
I want to be able to make them invisible and visible dynamically (at runtime) for a certain camera without turning them visible/invisible for all cameras.

Obviously Culling Masks and Layers should come in handy, but they don’t, here’s the Plot Twist:
You can only assign one layer to a Game Object. I can’t make a game object visible on multiple layers. So I can’t just have cameras that render only the default layer plus a specific layer assigned to them and then modify the Layer Mask of the “Toggleable GOs” every time I want to make them (in)visible, because in Unity there’s no Layer Mask for GOs, there’s only a single Layer property.
Doing it the other way around (assigning “Toggleable GOs” to specific layers, and modifying the culling masks of the cams) doesn’t work because of the layer limit. Keep in mind I might have more than 32 “Toggleable GOs”.

Then just turning them visible/invisile for each camera.