Help in understanding my network log (and help solve a server hanging issue)

I’m trying to improve the networking in my game (online first person shooter) so I’ve been studying my network logs to know what exactly is going on. One major problem is that sometimes when someone joins, the server will hang for a while – this can cause problems for everyone in the game. I have no idea what causes it and I’ve never been able to reproduce it using my own computers… when it hangs, it says ‘not responding’ (though it usually recovers fine) and it seems to stop any state sync updates but at least some RPCs are still getting through (like chat stuff, kills, etc – I haven’t checked if 100% of the RPCs get through).

The following snippet occurred when another player joined (so 2 people are playing with each other) –

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

Scope index 0 is now in scope for AllocatedID: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 34 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 35 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 10 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 9 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 44 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 48 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 45 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for SceneID: 49 Level Prefix: 0
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 1
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 2
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 3
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 4
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 5
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 6
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 7
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 8
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 9
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 10
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 11
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 12
 
(Filename: ..\..\Runtime\Network\NetworkManager.cpp Line: 2445)

New scope index 0 is now in scope for AllocatedID: 13

What’s it doing? How heavy is this traffic? What would I do to optimize it? It seems that at least some of it is redundant – like doing “Scope index 0 is now in scope for AllocatedID: 0” like 8 times.

And I know what AllocatedID is, but what’s the SceneID and level prefix stuff about?

Appreciate any help – I may post more log snippets later as I try to nail down what is causing the server hangs.

edit:
More questions:

What’s the difference between “Mode 1” and “Mode 2” for RPCs?

Any way to track traffic network volume? Like a linegraph or something showing how much data is going through? I only have indie (is it a Pro feature)?

I know that only the server accurately keeps a list of NetworkPlayers – would it be feasible to have the server broadcast that list to all the clients so everybody knows the right NetworkPlayers?

Is there perhaps a way for the server to detect that itself is hanging? Would help greatly for troubleshooting if I could print those events…

The scene ID, I don’t know (or remember), but the level prefix : http://forum.unity3d.com/threads/56574-Network.Instantiate-and-RPC-functions-theirs-magic

Makes sense thanks.

Anyone know about the other stuff?

[bump] Still wondering exactly what the Scope index 0 is now in scope for AllocatedID: 0 stuff is all about

Are you using a lot of network.instantiates? if so, they pile up in the buffer unless you clear it out. This would probably be the cause of your hanging.

Actually that was my first assumption about the hanging as well. I’ve eliminated nearly all buffered RPCs and network.instantiates (only ones I left were for spawning player characters because I can easily remove those from the buffer when they disconnect) . If the hanging is gone this next playtest (Sunday) I may know what was causing it.