.
I have been actively developing a multiplayer Unity game for over two years, but have run into showstopping issues with every aspect of Unity’s builtin networking library:- Every multiplayer action game I have seen that utilizes Unity’s built in networking is afflicted by one or more of these issues.
Although Larus has provided me with test Master Server builds to attempt to circumvent one of these issues, to my knowledge, other than this, these issues have never been officially acknowledged - let alone addressed - by Unity.
Here’s a representation of various networking discussions I have engaged in on the Unity forums:
I am now desperate to see one of these two things:- A full featured multiplayer action game (including serialized NetworkViews, buffered RPCs, interpolation, networked physics, etc) that uses Unity’s networking and that is not crippled by these bugs. If there is just a bunch of things that I am doing wrong, I would love to know what they are.
An acknowledgment by Unity that the built in networking is currently unsound, and a statement that they are working to fix this asap.
If neither of the above conditions can be met, I need to find a third party networking system that can serve in the role that Unity’s built in library is supposed to fill - as efficiently and cost effectively as possible.
When in the course of human events it becomes necessary for an integral component of a near perfect system to be abandoned by one of the system’s most loyal adherents; a decent respect to the opinions of mankind requires that the causes which impel the separation be enumerated before a candid world:
The current Master Server occasionally drops games so fast that they flicker in and out of existence even when they are registered every couple seconds.
The latest beta Master Server is incredibly unreliable. I have tested the Unity hosted beta server, hosted several versions of the beta server on my own dedicated server, and am now connecting my game to DimeRocker’s server. All suffer from completely random (and just as debilitating) instances where they can’t be connected to for anywhere from a couple seconds to hours on end. Master Servers can be connected to more reliably from native apps than from web players, and I have experienced instances where one client instance could connect to the master server while another couldn’t, so I am currently suspecting that this is a client side issue.
The Connection Tester is completely inconsistent. I had to give up on using the Connection Tester in my game because I got tired of it randomly telling me that I was offline or unconnectable. Does it work for anyone, or should it just be fixed or scrapped?
I have experienced many instances where game servers could be successfully connected to after the 2nd, 5th, or even dozenth failed connection attempt. These obviously aren’t network configuration issues - so why couldn’t the servers be connected to on the first attempt?
My game is now suffering from a bug that seems to grow more prevalent with each Unity release: NetworkViews are set up properly, and transfer RPCs perfectly - but they absolutely will not serialize with each other. This occurs most often in games that have been running for at least an hour or so, although it can happen at any time.
The networking contains a bunch of general bugs and oddities such as Network.Instantiate buffering it’s RPCs but Network.Destroy not clearing them and undocumented NetworkMessageInfo.Timestamp behavior. I am sure we as a community could compile a couple more if there was an opportunity of them being addressed
My game has an active fan base that would love to help with troubleshooting. Lets squish these bugs once and for all!
Half the problems you mention sounds like you rely on UTs hosted services (for connection tester / nat facilitator), which you shouldn’t as they are not for production use and have no guaranteed availability at all.
I’ve been using the dimerocker master + nat + connection tester for over a week with 10 parallel headless clients that run as server, and never had any problems to connect to any of them. The only thing I do aside of relying on unity’s own functionality is a remove - add host call to the masterserver because the automatic one does not seem to work reliably.
I run my own master server reliably and I have helped others to set up master servers on hosted virtual servers as windows service now, been ironing out kinks in my service code but all in all, very positive results.
Send support your issues, maybe they can help you resolve them.
I know of a few other people using my master server daily and they don’t complain or have any issues that I am aware of, and that particular test server isn’t running as a service as of yet, adding more trouble shooting text code into my service before I deploy it up to replace what is their.
in my multiplayer game i host my own masterserver too, but i have to restart my gameserver at least once a day (at night per batchscript).
otherwise the server would lose connection to masterserver and disappear or get strange physics behaviour or networking errors (“networkview id not found strange behaviour could occur”, players dont move on other clients etc)
i am using rpc (non buffered), unreliable statesync. and players network.instantiate themselves (not by server)
i thought about allocating the networkview ids manually, but i read about strange workarounds and bugs with it, so i kept my network.instantiates.
so most of the time i go fine with this configuration until more players connect and disconnect to the same time, which causes ugly bugs sometimes, which are very frustrating.
@Dreamora, Zumwalt;
I haven’t utilized Unity’s hosted servers in months - so that certainly isn’t the problem. Although not as frequent as when I was hosting a master server myself, I have been experiencing connectivity issues to DimeRocker’s server - even over the last couple days. I am very excited at the possibility of accessing a reliable master server, and will greatly appreciate all the information you can give me on what is special about your setups. I have gone back and fourth with support over the master server issue for months, but as yet, still am experiencing issues with every master server I can find. I have 11 open support tickets right now (and many more that were robotically closed without a resolution for 2.5.1), so I am kind of discouraged on the whole support ticket process
I have just decided to sign up for a 11 Windows VPS, and give one more attempt to host my own master server. (My last attempt was running on a decently loaded webserver, so who knows - perhaps that had something to do with it’s pathetic connectivity). I will also be building a dedicated headless server version of my game to host on the VPS and see if it helps to alleviate issue 5 enumerated above (which is by far the most serious to me anyway).
@Ethan;
When I figure out how to work with a master server reliably, I’ll let you know! Your Master Server’s availability shouldn’t have any correlation at all with your game’s physics or in game networking. You should definitely be allocating NetworkViewIDs manually - I was able to solve several annoying and hard to track bugs by eventually taking this route in my game.
Shoot me an PM with your ip / ms fs port info and tonight when I get home from work I will build you up a ms to run as a service and shoot you a downlod link for it, also with 1and1 make sou you add 4 rules to the windows firewall, they would be 2 tcp rules for the ms/fs ports and 2 rules for the udp ports let me know if you need and info or help and I will gladly type up a document on all of this.
always do, FYI, I’ll be giving you full code to the project also, so you can tweak it as you want, I just include the build with it, both debug and release along with installation instructions (command line you use and how you use it) to make it a service. I had offered it up to Unity but they don’t seem to have a want or need for this build of mine, which is totally fine and cool by me, I have found that through the logs, 1and1 do a maintenance reboot at around 4AM CST to 5AM CST depending on which bank you are on for the virtual servers and the server outage amounts to 4 minutes in total, with this build you will be getting, and running it as a service, you can track service outages.
It is up and running the rest of the 23 hours and 56 minutes though. The reboot doesn’t happen on every night, just when there are critical windows server updates to push out. As I stated though, I will help support you through the roll out and make sure you get up and running smoothly with the latest build. Make sure you have VS 2003 if you want to rebuild it, because upgrading the project to 2005 or 2008 also causes inherent issues with the master server do to changes in the Microsoft Framework and compilers.
I will get you that build and link in a few hours with full instructions on how to add it and run it on your Virtual Server.
I think its time for me to jump in here again to give some data on my ongoing tests with the Unity Headless Client of 2.6:
Number of Servers: I have 10 instances running the network example (modified to run optimal as headless clients and additional code to detect if a port is already in use as well as a few lines of code to handle master server updating)
Current uptime: The servers have been started 13 days ago. Neither did the server crash or require restarts during that time nor did they ever vanish from the Master Server Listing (kudos go to dimeRocker, I’m using their Master + NAT + Facilitator setup for this test. I would assume Zumwalt’s would work too)
Required CPU resources: I’m only connecting to a few selected servers to get usefull and representatitive data on idle cpu requirements vs in use requirements.
Idle Server: Use 1s of cpu time per day Used Server: Use 2-3s of cpu time per day (I’m spending 5-10mins all 2-3 days on them. The levels are the heavier ones means the buggy fun level and 3rd person which does position sync)
Server: Is my personal homeserver which is a Core 2 Duo E6600, Windows Server 2008 Standard, 4GB RAM. Connected to the web through my regular vdsl home connection (20’000 / 5’000)
I hope this data will help others to decide if unity networking is for them or to find out if their own code is the cause, which I’m sure is the cause in various cases.
As far as I am aware, Robur has been up and running since deploying my solution, last I heard from him, only 1 hiccup but that seemed client related and not master server related. Other than maintenance reboots by my ISP, my server has been up constant for 2+ months, and another company in CA that is using my service code has had a server up and running for at least a week as a service (when I finished the sevice code, would have to look again at the logs) and they have been doing testing with very good results.
The master is a bridge, I can’t seem to stress that enough, however, there does seem to be some sort of flaw or issue if you run it as a windows console application that doesn’t seem to exist when running it as a windows service. Granted the server that is running in each of these 3 cases mentioned are 8 way processor machines with 1 gig guaranteed ram, up to 2 gig burst ram. Honestly I never see the processor over 5% utilization with the masters.
(I don’t have access to Roburs master so he would have to verify the service stats), plus my version uses the logging feature and stat feature in the logs, so I am pinging the stats every 10 seconds and rotating the log with the default 5 meg feature, this way if anyone who is using my service build has an issue, they can simply zip up the log files and email them to Unity support as is and Larus can easily read them.
Zumwalt’s master server build running on a dedicated 11 vps has been performing flawlessly for me around a week now with an average load of 60 or so active players @ any given time, with the exception of an occasional crash-inducing buffer overflow which he is working to isolate.
I have created testcase Unity projects which demonstrate the Network.Destroy (and much more important) NetworkView Serialization bugs, and they are in Larus’s hands. Larus is also working on a number of other bugs which Leepo has assembled into a list.
I completed coding the dedicated headless server for my game last night, and will be conducting performance testing. So far I haven’t been able to successfully get multiple instances of my headless server to run properly side by side on the vps, but this is probably just a minor thing I did wrong somewhere. Dreamora: Does your “additional code to detect if a port is already in use” look anything like this, or am I on the wrong track altogether?
MasterServer.dedicatedServer = true;
Network.useNat = !Network.HavePublicAddress();
var port = 2500;
while(true) {
info = Network.InitializeServer(10, port);
if(port < 2600 info != NetworkConnectionError.NoError) port++;
else break;
}
if(info != NetworkConnectionError.NoError) Debug.Log(info);
else Debug.Log("Server Running on port " + port);
I can’t wait to see how my new dedicated server setup will work in the wild! It looks like this situation may have a positive derivative after all.
Interestingly enough, when hosting the headless server and two client instances locally on my dev machine, I was never able to achieve less than 90 - 100 ms of latency on serialized packets sent from one client instance to the other. Do “serious” fps games have the same kind of networking overhead - and if not, what are they doing differently?
I run 2 - 3 headless servers on one machine, and it works fine for me. I duplicated the gamefolder to change the config (servername, port…) and the games exe file name.
Just to give you a number:
My game uses unreliable and it has no problems at all with the latency. A good connection has about ~20-30 ms latency (well 2 x ping) over the internet.
Did you try unreliable for your timecritic stuff?
I was able to get multiple master servers running on the same vps by instructing them to each utilize a random port between 2500 and 5000. My previously posted init code, which started at 2500 and incremented the port number with each successive init fail, (and which worked perfectly on my dev machine), didn’t seem to work at all on the server. The second server would start on port #2500 as well, and both would immediately disappear from the master server.
You have to be very careful when measuring timestamps: the NetworkMessageInfo.timestamp variable is based upon the send time of that exact network packet. When a client sends a serialized NetworkView to the server, that packet is not forward directly to other clients. The server’s NetworkView merely updates itself with the data, and then sends the data to other clients on the next NetworkView serialization. To accurately measure lag, you have to add some logic to the server to preserve the timestamp of the original client’s send time. Here’s a script I wrote which (I believe) implements this correctly: http://www.unifycommunity.com/wiki/index.php?title=NetworkView_Position_Sync
I am using unreliable NetworkViews exclusively in Mars Explorer.
Your main problem likely was that you ignore basic network development rules.
Generally all port in the 1 to 9999 range count as “reserved” as various services expect various ports to work or will bite you.
I think if you use ports in the 10000+ range for your initialization, there shouldn’t be a problem.
I think the random port also is kind of troublesome for clients to use the right port. if they are just incremented, its at least a predictable pattern by which they can search
I’ll try beginning @ 10,000 - why doesn’t Unity’s networking example do this?
The Master Server stores which port a server is on, so connecting to games hasn’t been a problem at all.
I am still trying to isolate the buffer overflow, the error that is in the logs on the server point to a strlen (string length) return that when a string copy happens, data is going beyond the real length of the string or some data seems to be “locked” by something, as of yet undetermined. I have now went as far as buying a more powerful PC laptop to isolate the problem locally since time between cycles from my desktop to my VPS possibly is also suggesting that the stream of data might have enough packets dropped to confuse the engine (> 10%), I am wondering if the header of the packet that causes the crash is fully intact but the buffered stream is incomplete.
This should not crash the master server though, so this is taking longer than expected to track down.
I am dumping large amounts of serialized information into the packets to test for “worse case” and trying to find the hole. I have a tard load of data going back and forth from the server to the client through the master and I have tons of debugging information. I am trying to isolate a repeatable scenario that I can fully document all of the conditions to make the problem reproducable to at least an 92% success rate.
chuckle
Ports 1 to 1024 are reserved on all (normal) operating systems, for known and default services. All ports above 1024 should be valid and available.
