Making Unity into a powerful seamless MMO engine - behind the scenes

News & General Discussion
1

As you might have noticed, we have made:

  1. uLink - a network integration showcasing the future of network development
  2. Pikko Server - a server layer that scales Unity servers into a dynamic cluster
  3. Tank vs Robots - a browser 1000-player FPS game.

For most people, we might seem to have popped up from nowhere so I thought I’d take this opportunity to explain a little bit about who we are and why we did such a crazy thing. And to take the opportunity to mention the main technological challenges we encountered.

As it happened, our team got the chance to spend a few years on whatever we wanted to do and we decided that we wanted to change the way networking for games was done.

Network development in games used to be a disaster from so many angles and it suffered from both technical and organisational problems. It was very unforgiving to be a network programmer and the task did not work well with the “crunch n’ patch”-development culture. In this stressful environment, your tools need to be just as your best friends: reliable, helpful and direct. They should be able to handle that you happen to have a bad day, support you when you get that crazy idea you want to show the world and they should just be there, making those boring everyday chores joyful.

We had been talking to CCP about a great way to demonstrate what our group thought would become the future of network and server software technology. We agreed that if we could retrofit Unity and make it into the most powerful MMO engine in the world - and show that you could play a fast paced MMOFPS-game with it, it would be the ultimate proof. Since there is a small but significant trade-off between a little more latency for a lot more CPU-power in the Pikko architecture, a massive FPS seemed like a great idea.

uLink

We already had a small crush on Unity long before we started developing these products. However, we thought that it was only half of an amazing engine since it had very rudimentary networking capabilities. At first we didn’t think we would build a complete network integration but after learning that Unity was pushing network developers out of their editor, we realized that something needed to be done about it. That something was uLink.

The more we worked with uLink, the more we fell in love. First with uLink itself, then with Unity and then with uLink, again. We built a special network torture chamber for it (using the automated testing tool Tsung), in which its every little aspect got hammered to the extreme, over many months. We put it in the hands of 3D artists with no real network experience as well as in the hands of veteran network programmers. The more we abused uLink, the closer to perfection it became. I would be very surprised to hear someone do something horrible to uLink that we had not already done in cubic-metrical-dimensions worse. We simply wanted uLink to be that friend that you always could count on. So we tortured it like a sadistic little kid.

But having built a network integration from the ground up was necessary because we needed to know every little row of code and every little package in the network in order to be able to do what we were about to do next.

The road to Pikko Server

We come from a pretty small Swedish town, Uppsala, that is most famous for its university. Our professors decides who will get the Nobel Prize but in general nothing really exiting happen here. Surprisingly, Uppsala University had become the scene of a small but stubbornly growing community of telecom programmers that used and developed Erlang. Erlang is a functional programming language originating from Ericsson and it has some truly amazing properties for networking. It is scalable, concurrent and very reliable. We dreamt of making a game engine’s server side behave as if it was programmed in this network optimized programming language.

We were not alone in thinking along these lines. Our friends at Artplant went for the straightforward approach when they made the Battlestar Galactica MMO with Unity as the client, and their entire server side in Erlang. I met their CEO, Jack, not long ago and he told me that they had not had one (1) server crash since they launched. He smiled. I smiled. He smiled more. Then we said the word we were both thinking out loud together. “Erlang”. (I imagine there are quite a few guys reading this and nodding in recognition of the similarities between the Erlang community and a religious sect). Even though Battlestar Galactica is a great game, and even if we have been helping out Artplant a (very) little bit with both the database and software design issues for their game, we do not believe that game developers should be forced to learn functional programming and have to throw away all the hard and excellent work that have already been put into their game engine. One solution might had been to make an object oriented script language that extended into functional scalable code. This would open up, in theory, that you could port game code back and forth.

RocketScript

Our team made shock waves within the Erlang community, when we were the first to have made a complete object oriented language, RocketScript, that extended into functional Erlang. It had been something of a Holy Grail for a group of code hipsters and academics that saw it as a way to popularize functional programming. However, we also saw it for what it was, and we actively discouraged people to use shortcuts such as RocketScript, as it turns out to be an awful way to produce functional code. This generated even more shock waves. People saw us throw away their Holy Grail, but it was a well grounded decision. RocketScript was simply not the way forward to make the future of network games. The still unmade games of the future deserved better.

Being a tool

I often bore my surroundings by regularly finding reasons to tell them that we as humans are defined by our tools. You and me are genetically the same as our cave crawling ancestors but as we look at our civilization and our human capability today, it is all there thanks to our tools. It is usually at this point that the other people in my team starts hiding away the red wine and clear the room of fragile minds and stuff. What if making tools made you into a tool? Or even stranger, what if you could take the idea of a tool and then use that tool in such a way that you could change one tool into another tool? Well, it doesn’t make that much sense but this is actually what we did with Pikko Server. We took the idea of Erlang and made a server layer that made the rest of the server side behave as if it was all done in Erlang even though the game programmer didn’t have to type one single row of functional code.

But we didn’t stop there. No, we had just opened the door into something that held much more potential than simply solving scalability over the CPU. We could now deal directly with the traffic between the client and the server, which enabled us take a broad and serious grip on addressing bandwidth management, network LODing and other optimizations.

One thing is many things

We have of course done many brilliant and stupid things along the way that would most likely deserve to be mentioned, but really, most things are better experienced rather than talked about. Still, one thing that have made people understand the potential in the technology we have been working on, is worth mentioning. That is our load balancing simulator demo MastSim.

I’ll post a video about it later.

A world record

We made an early prototype of Tanks vs Robots working with Pikko Server and Unity, pretty fast. But at the very moment we had made it we understood that it was pretty pointless. Of course, we had made an MMOFPS and it worked just as we wanted it to do and that was pretty awesome. But it was a little bit like being the first to climb Mount Everest and finding out that all the people you wanted to celebrate it with was still down at base camp. Or at home. So we either needed to spend a lot of time explaining how wonderful it was on top of the highest mountain in the world - and still not be able to make the feeling justice. Or we could build an escalator up to the mountain top and make it so damn easy to be there with us, and we all could party together like furry animals! You already know what we did…

The world record game is not a simple PR-gimmick but the last step in a long series of tests that defines the new frontier of MMO technology. We have added a lot of busy sleeps in the server and we have bloated player states just to make sure that this is as far from an ideal lab experiment as possible. So, without these extra loads, we could of course handle a lot more players, but then it would have said very little about how a typical game would behave together with Pikko Server. I guess I am trying to say that Tanks vs Robots does not demonstrate the outer limit of our technology, but rather defining its starting point. If you want, you can take part in changing how MMO works. Our failure or success will only be a few clicks away in your browser.

…

2

Here is a perfect place to mention how generous Jacky Mallett at CCP was and how she helped us polish the framework and the theoretical foundation of Pikko Server. Because it often doesn’t matter how great your technology is if no one can understand it but yourself. She is one of the greatest engineers we have encountered in our professional lives and we have been very fortunate to have got to know her.

We have made a lot of effort in trying to educate our users in how our tools work but even more time have been spent on making the tools so usable that they often do not need much explaining at all. For example, if a Unity developer would like to make an MMOFPS such as Tanks vs Robots, all the developer needs to do differently is handle mainly four new RPCs. There is no reason why performance and usability are so often pitted against each other. Why choose when you can have it all?

…

This post became much longer than I expected. If you made it all the way down here, you better let me know what you think with a comment. I will gladly answer any questions even though I might be a bit slow since there are a lot of things that needs my fingertips. Eventually, I will answer them all - don’t worry.

Christian Lönnholm, CEO Pikkotekk / Unity Park

3

So >.> I’d like to help you out, but you forgot to link to this mmofps which sounds pretty awesome. These products you’ve developed are so neat, I hope it goes a long way into continuing to make unity a standard fun easy to use game engine many of us have come to enjoy working with.

4

Thanks for reminding me! I edited some links in there now but here is the homepage for the game:
http://www.muchdifferent.com/?page=game-world-record

And yes, Unity is awesome. =)

5

woot erlang is a great language that are mostly unknown to the masses, i have to agree at first functional languages scares me aways till now lol, i don’t have the courage to continuing study it, my intention was just writing simple demo server on erlang after i read an article on gamedev, other problem was i don’t find ease to use tools to compile erlang code into a bytecode or did i missed something?, anyway if your product become succeed will you sell it? at what price range?, how about the server source code? i assume you also haven’t found the solution for erlang bytecode?.

6

Ok, you’ve got my attention :slight_smile:

So your solution has both client and server elements, with the client side capturing the Unity network IO (presumably then expanding on the limited Unity Mono runtime, using the erlang VM, language, and incremental GC (et al). and handling all the server side as well as the client side IO, sync, etc. Is that correct?

I’d be interested in knowing how the erlang vm differs from “realtime” Java VM, but perhaps that’s a question for Ericsson.

When you say you automatically handle MMO/Network, do you mean you automagically detect the Unity runtime and insert your solution between Unity and the native IO?

what platforms does your solution run on?

hehe, these and many more questions :slight_smile:

Cheers,

Galen

7

Hi Stellar!

To compile it into bytecode you could for example use the module “compile” or the erlc binary. We simply use a makefile that gives the compiler a list of src files to compile but you can find everything you need here.

We are fortunate enough to already have the interest of a few companies wanting to buy our Pikko Server. The price is negotiated and depends on a number of factors but I guess that there is no way around the fact that it is very expensive. But we have more innovative technologies coming soon that is much more affordable. You will be able to find everything we offer here: www.unitypark3d.com

The Pikko Server code is normally run and hosted by one of our hosting partners together with the game deployment. We make exceptions to this sometimes but this is the normal case of how we go about installing Pikko Server.

I didn’t understand the last part of you comment about a solution for Erlang bytecode. Would you mind trying to explain it to me one more time?

8

Galent,

Actually, Pikko Server works much smoother then you think it does. It works as an Application Level Gateway (ALG) on the server side only, where it reads the uLink protocol and then do some fancy magic that is explained here. Basically, what it does is that it divides the virtual world between different Unity servers (cell servers) and then return the result to the clients as if it was only one Unity server that the clients was connected to.

All of our technology support all the same platforms that Unity support.

If you have more questions - just fire away!

9

Who’s teeth did you have to pull to get the packet definition from UT? I have been trying to get them to release the packet definition information for 2 years now so that I could make something native to support a back end server and load balancer, at some point I have an email that it will eventually be released as soon as it is fully documented, but that was sometime last year before 3.x was released. That packet structure is key to solving many issues including setting up a full blown stand alone server interface, I guess it takes a lot of power or ‘know the right people’ to get that information? Or did it simply boil down to $$$

10

Dear zumwalt,

We didn’t use Unity’s packet definition. We built our own network integration, uLink, from the ground up. You can learn more about it here.

You seem to have had pretty cool ideas that looks like a bit like what we have done. Care to tell me more?

11

@Christian_Lonnholm - ok, so basically you use the native Unity networking, but replace the server side with a clusterable (vertically and horizontally) custom server. Fair enough. The only thing I’d be missing is decoupling the Unity game from the network IO on the client side (again, because of some Unity implementation issues), just to speed processing on systems where dual processors and memory space above 32bit are available. As for game dev using Unity, I like it :slight_smile:

@zumwalt - I see 2 possibilities:

  1. they packet sniffed out the details
  2. there is much to be said for being able to pile a couple of cases of Polish Vodka into an unmarked van and hanging out in the pub near the Unity headquarters :slight_smile:

Cheers,

Galen

12

Another question arises, so … that works fine when the clients do conflict resolution, etc… as in the best way to leverage the built-in Unity networking. But how’s that going to work with server side physics, conflict resolution, etc… am I able to couple a Unity server implementation with the cells?

Thanks,

Galen

13

Hmm, I think the doc just answered my last question… really interesting read (I really should finish to see who did it :), but it creates some interesting questions). Cells look like a grid computational model, with the Pikko server doing many of the things I see in virtualized environments (like VM ware). How are the virual worlds mapped so you can do the overlays (ie cell tower analogy)? Is it straight Vector3 mapping/gridding or some other mechanism?

Sorry for so many questions… new shinny thing syndrom I’m afraid :slight_smile:

Thanks,

Galen

14

This looks interesting :stuck_out_tongue:

15

Galen,

Just to clearify, we have made uLink, our own network integration in Unity so Pikko Server does not work with Unity’s native network. I realize that I have been unclear about that. The Unity/Raknet integration is something we are happy that we have stayed away from. Even though we made our API pretty much the same so that it would be easy to move projects and so that we didn’t have to do so much documentation in the beginning.

We actually had uLink multi-threaded until 3.X when we got better performance out of it as thread locked.

If you have seen us hanging with cases of vodka outside Unity’s HQ - then that was just because we happen to be there by chance and we had no idea that it was vodka in those cases. But if you didn’t see us there and was just speculating - then I can tell you that we would never try to bribe people at Unity. =D

16

Galen,

I missed your last questions as I was answering the earlier one. Well, we have made a new world partitioning system called “the Mast Algorithm” which allow the cell servers (that are shaped in a hexagon fashion) to adjust its size and even move with players if that seem to be the most optimal way to balance the load. I have already promised to post a movie that explains it much better then I can do in words… Anyhow, the Mast Algorithm make the server side behave very organically to player activity and recalculate the most optimal world partitioning five times every second.

It is great with questions! Keep them coming.

17

Looking at your uLink Manual that you have posted, but the bottom line is that Unity with RakNet utilizes a single instance server / client from the start, what they did is really straight forward. Jenkins code is setup so that you can run multiple instances of the client talking to multiple instances of a server or load balanced as it were. All UT did was start a RakNet static instance before the client starts, from what I can tell, 2 of them actually, one as the server and one as a client. The downfall is that the client can only “connect” to a single server instance, you can create an array of clients[×] on start and have an array of servers[×] on start natively but not within Unity. I actually had a request back when they first setup RakNet to work with Unity to do just that, but it never happened.

So then, they have a structure setup that they place all objects in, so the broadcast is the entire serialized data structure of which ever viewed object it is attached to, the thing is, that structure definition is unknown, the thought is, switching up the client.connect to talk to a full blown raknet server, not just another client that is running as the server and be able to do stuff with that information. Simplified discussion of this topic only, but say I write a raknet cluster, if I had a client.connect[member#] (not allowed in Unity at the moment) if server A disappears, server B already knows about what is going on and the player continues and I can deal with the information broadcast to other players or store in a database or whatever I wanted to do.

This goes further though, since programmers can only do a client.connect(single instance), having my own game server, or business logic server, better yet a traffic cop, I see the player come in, look at available shards that I have created and look for the least busy server and attach the player through the traffic cop to that server and that server is a ghost client to all the other servers so all players know about each other regardless of shard (ok my term, but shard is just a member server in the bank), I can only do this if I have the packet structure, at the moment anyway.

I have already done this with native raknet in the past but not with Unity Raknet implementation. Problem at that time was that the player would randomly disconnect from the server during a game loop, never could figure out what was going on, on top of that, my code used custom plugins I wrote which as you know do not work on phones or web, but native Unity with RakNet does work across platform, thus my dilemma.
The network loop resides outside of the game loop, raknet is running before the client game loop begins, it has to work that way for a solid state connection and on each game loop they control the heartbeat to the server. As far as I can tell, they are using delegates to pass how many connections the server should accept out to the network thread and they are running raknet single threaded when it can run multithreaded, to add pain to punishment, I used Intel’s parallel processing routines to load balance across CPU’s and hyper threading. I wanted to add scalability not only to the server but to the player, another point in that process was to allow for Amazon Cloud along with 1AND1 Cloud to be able to instantly scale based on a cloud change in processes.

So if the load in the cloud increased, with the parallel processing plugin going, it immediately and seamlessly took advantage of the new processing power which ultimately allowed for additional connections.

Which was the reason why I purchased into Photon by ExitGames, it is very powerful, straight forward, scalable, although manual reads like stereo instructions, has a strong support staff with very knowledgeable employees.

You see, there is ultimately a dilemma, all other forms of networking happen AFTER the client has started, all from a script object in code which has one apparent problem, every switch from level to level, you have to reconnect the client, if the networking server was started prior to the game starting, the game then is waiting on which position in the stack that the network component lives prior to start which is a problem because game objects could easily exist prior to the network interface unless you code to avoid that problem with a global ready state and manually instantiate all objects past that point.

The only “fix” to this was to create a single script master EGO that controlled all other scripts, then on each game loop update validate network state and move on, which tends to lend to lag. Otherwise you are stuck with starting your binary to begin your network interface first, then having a plugin inside the game project that talked to that alternate program through p-invoke or native call, then you have a 2 part problem, is that running and is it monitored, and is your game able to talk to it without issue.

Ultimately the best solution is to begin network code, then begin game code, after game code terminates, terminate network code.
Ya, that is the simple high level conversation from the 10,000 foot level.

Thus I thought you had talked someone at UT into giving you the packet definition so that the native interface could be used with other entities. I would rather use the bitstream interface for raknet anyway instead of the bytestream.

18

Wow looks like you guys really have some hardcore stuff going on. Forgive me if I missed this though… but you’ve posted it in the Unity forums (primarily hobbyist with some indies kicking around here), I’m just curious if you guys are planning on somehow bringing this technology to the hobbyist-indie game developer and what those plans may be?

Regardless you guys have kicked some serious butt here and the technology looks interesting! Great work.

19

zumwalt,

I feel your pain. Accessing the ability to parallelize is a cause worth while though. As you mention, you get very nice features if you manage to cluster your server side with redundancy being an important one. I hope that you can draw inspiration from how we solved it with Pikko Server. It is hard to give more specific feedback since I know very little of exactly how you are going about this problem but please e-mail me and my team will take a closer look at it and see what we could help with.

On a side note, I was really happy to hear that you are impressed by Photon and the team over at Exit Games. I have met them a few times and they have been working hard on getting their solution to where it is at. Even though we are, at some level, competitors, I have the deepest respect for the people making tools for network programmers. This respect extends, of course, to you too zumwalt.

20

Thanks DavidB!

We are surely putting our technology in the hands of indie guys. We have already given indie developers about 200 000 € worth of licences and we are constantly looking into how we can help out even more. Though, this is done as we get the opportunity to do so, as we think it is important to make sure that people don’t waste their interest and talent (= lives) on solving problems that have already been solved a million times. It hurts deep inside me when I think about the inefficiency…

Thanks again for the kind words!