I am trying to get information from my database, through the server, and onto the client. This works, but the problem is that I cannot operate on this data. The data comes back in several packets, and one packet is insufficient to operate on. But the function I call to operate on the data attempts to operate on the each packet individually, so the program stalls. I know async might help, but as I understand ReceiveCallback will never stop going because it is always listening for more data.
private void ReceiveCallback(IAsyncResult AR)
{
//Check how much bytes are recieved and call EndRecieve to finalize handshake
int recieved = _clientSocket.EndReceive(AR);
if(recieved <= 0)
return;
//Copy the recieved data into new buffer , to avoid null bytes
byte[] recData = new byte[recieved];
Debug.Log("RECIEVED " + recieved);
Buffer.BlockCopy(_recieveBuffer,0,recData,0,recieved);
//Process data here the way you want , all your bytes will be stored in recData
Debug.Log("Recieved data " + recData);
string toDeserialize = System.Text.Encoding.UTF8.GetString(recData); //Convert byte data to string
massiveStringFromReports += toDeserialize; //Put all packets into one string, this is what I want to operate on
Debug.Log("toDeserialize: " + toDeserialize);
Debug.Log("Massive String: " + massiveStringFromReports);
Painter.drawing testing = new Painter.drawing("null", "drawingID", "geohash", "sessionID", filler, "endOfDocument");
string StrtoDeserialize = setupDrawingObject(); //This function reformats massiveString so it is appropriate and can be further processed
Debug.Log("SECOND TO LAST CHAR SHOULD BE ~: " + StrtoDeserialize[StrtoDeserialize.Length - 3]);
//If this is ~, then do rest of stuff, aka call the function on 257
if(StrtoDeserialize[StrtoDeserialize.Length - 3] == '~'){ //If it ends in ~, it's a full report, so proceed with operating on data
testing = DeserializeString(StrtoDeserialize); //deserialize the string into an object. It stops here! I know the string is valid because I can deserialize it in a dummy file.
Debug.Log("TESTING'S GEOHASH: " + testing.geohash);
}
}
My main question is this: How can I wait until all data is received, but not have it wait indefinitely, and then call a function that operates on this data?
I tried to include all necessary info without extra, but if you need more let me know!
Generally you have some type of manifest of what you need. “I need parts A, B, C, and D before I can continue.”
Something is in charge of receiving the data and checking if all parts are present.
When all parts are present, fire off whatever process is next and needs A,B,C,D before it can happen.
Also when you start, start a timer that when it expires, it declares that a timeout has occurred and tells the manifest watcher “give up, it’s over.”
Obviously, when the manifest watcher is satisfied, it should also tell the timer to stop.
That is what I am attempting to do on line 25. If the big string with my report ends in the special character ~, then I want to call the function. Are you saying this is the right strategy, but somehow I am executing it incorrectly?
Honesetly it’s pretty hard to reason about ad-hoc live string processing off a wire.
Since this is 2021 have you considered using any one of the myriad data transport protocols already out there and widely supported across many tools? For text transfers I’d just put it all in a JSON object and call it a day.
Actually, yes. I originally was using Newtonsoft and had a DeserializeObject<Painter.drawing> line, but it had this same issue. The string I get back from the server is canonical extended json (as opposed to plain json), which of course cannot be deserialized through the normal Newtonsoft command (as far as I know). So I have to process it to get rid of the many additional characters.
So i strippedthe characters off, but as I was trying to deserialize the string, it would attempt to deserialize half of a full report, which resulted in the same issue. So, I tried to deserialize by hand, and I ran into the same issue, which is when I realized what the problem truly is. I originally thought something was up with Newtonsoft.
Line 16 takes this chunk and appends it onto something… are these chunks guaranteed to come in order without gaps? That’s an important requirement obviously.
Either way, debugging a live datastream is a bit fraught and attaching the debugger or using Debug.Log() to reason about what is coming down and how your program is responding to it is pretty much the only reasonable step forward.
No, there are no gaps. ToDeserialize is a string of one packet, and I attach it to a larger string that is every packet by the end. I print them out and they are what I expect them to be… until I throw in a function that operates on them because the function is called on the first packet and the first packet alone. I am just trying to find a way to call this function once all the chunks have come through. I will rethink how I am doing it based off your first response.
Do you think async could be useful here? I don’t have much experience with it, and when I tried to use it I couldn’t return a Task<> without an error.
Just don’t call the “function that operates on them” when you receive the first packet. Have somewhere you store the packets while you are waiting to receive them all. Each time you receive one, instead of calling the function which operates on them, you instead check if you have received them all. If you have not, then you don’t call that function. If you have, then you do call that function.
Hello, thanks for chiming in. How do I know when all packets are received, though? The function never returns 0, presumably because it is always listening. Also, the received variable is never set to 0, which might signify no new packet has arrived. I’m stumped as how to keep track of when the last packet has arrived.
Where are these packets coming from? Aren’t you the one creating these packets? Isn’t this about your server to client communications?
If you’re creating them, I would add a header to each packet which includes a packet ID, a message ID, and how many packets make up this specific message.
The packet ID would be unique for every packet. You use it to detect and throw out duplicates. You also use it to sort packets into the order they were sent.
The message ID would be unique for each message, where a message can be contained in a single packet or multiple packets. After the message ID, you have the total number of packets in this message which lets you know how many packets you need to receive before processing this message. Once you have all the packets to reconstruct the message, you can sort them in order by packet ID if you aren’t doing so when they are initially received.
Once you have all packets received to reconstruct the message, you do so and call the function which processes the contained data. You can do all this just with simple lists if you want. You can have a list of received packets pending processing, and a list of messages which are waiting for additional packets. When you receive a packet, you check if it is a message needing multiple packets, if so you check the messages list. If it is there, you add this one to it, if not you add a new message to the messages list. When you receive the final packet of a message, you process the message and remove it from the list.
There’s a lot of other ways to build your own network transport than the above, but this way is a pretty simple approach as far as these things go.