I’m reading from a UDP stream where I know data should be coming in a certain order (e.g. there will be one double from bits 32-95 and another from bits 96-159). How can I go about parsing this? I know how to convert everything into one big string, but I was wondering if there was any existing helper methods to pulling out ints, doubles from a byte steam.
Thank you very much, sir!
If I want to grab individual doubles, do I just have to create multiple different arrays and convert them? I don’t see a version of this method with length as a parameter. Otherwise, how would it be able to differentiate a 2 3 from a 23 in a stream?
If you have a stream. You can read a set amount straight into that.
However, you might want to look into something like Protobuf to send data and serialize and deserialize it. If you have many different message types.
Nobody should be writing their own data formats in this day and age. Easy noob mistake to make, but yes use protocol buffers. It’s the best all around serialization to use for games for a whole bunch of reasons.
Can you elaborate on this? Advantages/disadvantages to using protocol buffers? why is it a noob mistake to not use protocol buffers?
It’s a noob mistake to re invent the wheel, rolling your own. But then the OP sounds like he was doing that with his networking also, so…
Protocol buffers is kind of the defacto standard for serialization in high performance server frameworks and networking libraries. Not necessarily in the game industry but just generally.
Protocol buffers is space efficient, which is a big deal in games. But the real power is the varint/msb encoding it uses for integers. Which it uses for bools/enums also. Msb encoding lets you encode integers using less then the number of bytes they normally take. It’s optimized for the kinds of small numbers games send a lot of over the wire. So combine that with the fact you can convert floats to integers over the wire to take advantage of this (and at the same time sending just the precision you need), and you have pretty much the optimal design for multiplayer games.
While your points are not bad. I personally don’t recommend Protobuf. It generates too much garbage IMO. (That is if you can’t send a stream object directly).
Unless you have found a sweet way to use Protobuf.NET without generating too much garbage. If so, I would be very happy if you shared it. It has caused us alot of issues so far.
Thanks for elaborating. I haven’t really used things like protobuf. I’m glad to see it has options for sending integers that use less bytes, but I do kind of have similar concerns as @TwoTen here - those libs add a bit of bloat.
In my own project, I wrote my own version of NetWriter that lets me pack ints up to a specific number of bits, not bytes. This is pretty much the most condensed way you can send messages over the wire. I’ll admit it’s probably more error prone than something like protobuff because you need to make sure you read from the stream in the exact way you wrote it. But overall it’s pretty lightweight. One class, and I have tons of control over when I allocate buffers.
Yeh, the biggest flaw IMO with Protobuf.NET is that we have to use Streams. So it’s a pain for the GC.
But we are currently using Protobuf until we find something better. But we have done our implementation easy to replace.
We us the RecyclableMemoryStream project that a was released by MS instead of using the MemoryStream. And we just have a helper class liket his:
using Microsoft.IO;
using ProtoBuf;
using System.IO;
public class Serialization
{
private static RecyclableMemoryStreamManager streamManager = new RecyclableMemoryStreamManager();
public static T Deserialize<T>(byte[] data) where T : struct
{
try
{
using (MemoryStream stream = streamManager.GetStream())
{
stream.Write(data, 0, data.Length);
return Serializer.Deserialize<T>(stream);
}
}
catch
{
throw;
}
}
public static byte[] Serialize<T>(T record) where T : struct
{
try
{
using (MemoryStream stream = streamManager.GetStream())
{
Serializer.Serialize(stream, record);
return stream.ToArray();
}
}
catch
{
throw;
}
}
}
So if we ever need to, we can just replace Protobuf.
GC has never been an issue for us. But the whole reason we use protocol buffers really is so we can leverage varints, so we just don’t send a lot of data even for hundreds of things moving around. So the GC of serialization is almost white noise compared to other stuff in this case. I would have to believe you are sending a lot of data you really don’t need to, to be seeing GC pressure that matters.
Outside of fairly uncommon scenarios, you should never be sending whole floats or vectors. We optimize most of our space out by just using movement techniques that either require inputs or just a 2d vector and a heading. Which works for 90% of what most games need to do.
For stuff that is updated often, like incoming movement updates which generally account for around 98% of what you are dealing with, there are straight forward ways to optimize that more.
First one is track on the server which players have seen what, and don’t send data if it hasn’t changed.
Second one is a big saver, send delta’s. So just send the amount moved not a complete position. It’s fairly simple to calculate what the maximum movement could be in this case. Which you need to do so the client can detect if a packet was missed. In this we we send a complete position once so the client can resync.
All of that combined and the bandwidth it takes to track 100 moving characters comes in at around 40kbps. Which is why our GC is white noise.
Also one more thing that can have a huge impact. Don’t send updates to clients as they happen to the clients. Let the update from the client be the trigger to grab everything in range from your spatial grid, and send that as a response. So at every network tick, the client sends and receives one message in each direction. This also leverages protocol buffer arrays better, you get more space savings.
Well, optimally, you want to avoid the Heap all together. If we allocate a few KB a frame, that adds up.
As for your optimization suggestions. We send our rotations as integers.
And we don’t do delta compression. Cause we send our movement on the Unreliable channel. But what we do is we mark our fields as required = false. Then we pass them to 0. And they are not included and the server knows to just use the same as last time
Forgot to be explicit about what I mean’t by whole floats/vectors. This is what we use for converting floats/ints and visa versa. It’s rare to need more then 2 points of precision. We have our own classes for 2/3/4 dimensional vectors that use integers, and those are what we send over the wire.
public static float ToFloat(this int i)
{
return i / 100f;
}
public static int ToInt(this float i)
{
return (int)(Math.Round(i * 100, 2));
}
[/CODE/
Compared to? You always have to ask that question in context. Otherwise it’s mostly meaningless. The GC created by the unity character controller for example, completely dwarfs serialization.
A few more bytes actually might not matter AT ALL. Because it’s what is pushing the collections to take place that matter. This idea that any extra bytes are bad needs to die in a fire. It’s extremely dependent on your specific game as to whether it matters and how much.
1 Like
Ye, you’re right man.
Also, do you use structs or classes for your objects?
Lolz you would ask that. I’m using classes because on the server we have some database stuff that can’t serialize structs. Ran into that recently so I just changed them to classes for now. Didn’t want to hassle with solving that at this point on a new game But normally it would be structs.
40kbps seems a bit high, even for that many characters. how often are you sending updates?
i did a quick calculation here for my game…i send actual 3-D (compressed) position & rotation data at 14 bytes per character snapshot (that’s including 4 wasteful bytes for a unet net id). with interpolation on the receiving end, it’s pretty easy to bring this down to 15 updates per second without a huge degrade in quality. so that’d be 21kbps for 100 characters. figure a little extra data sent with each message too (message id, character count)
not that i’ve actually pushed my system this far. i’m doing a 6 player vr game. so 3 transforms per player + maybe 5 other transforms that are always around, and then whatever else the players spawn during a match. so realistically 23-30 things at a time.
i really like talking about bandwidth optimization lol
1 Like