The maximum payload size is the length of the largest single message that can be sent through the transport. You may think of it as the maximum size of a single “packet” (although the transport may elect to fragment the payload into multiple actual network packets). We require this to be configured upfront to allocate buffers at initialization (we want to avoid dynamically allocating memory at runtime).
What influences its value is what the largest message Netcode for GameObjects (NGO) can end up sending through the transport. By far the largest message NGO will send is the message with the current world state it sends to newly-connected clients (other messages following this one are typically pretty small). The size of this message is influenced by how many network objects/variables/transforms you have in your scene.
The simple answer is whatever value makes your game work correctly.
The more complex answer is that it depends on how many objects/variables/transforms are being synced over the network. Unfortunately we don’t have any guidelines currently to map a number of objects to a maximum payload size (e.g. 100 network objects require 50KB payload size or something like that), so finding the ideal value involves some trial and error.
Note also that aside from using slightly more memory than necessary, there shouldn’t be any issues with setting this value too high. Erring on the side of having a value that’s higher than strictly required might be a good idea.