Questionable matchmaker behaviour leading to frustrations

Unity Services
, ,
1

Some observations about matchmaker and questionable behaviour that leads to very odd situations.

User is not required to keep their matchmaking ticket “alive”.

If a user queues for a particular mode, say a 1v1 and then immediately (before the matchmaker has assigned them to anything) quits their game or pulls the plug on the internet, there is no built in handling to invalidate their ticket (such as an inactivity timeout). This is bad for a few reasons:

  • If the player loses connection or their game terminates for any reason, (unless specifically handled) their matchmaking ticket will sit in the queue until another player matches with them. At which point the player they matched with will enter an empty match.
  • If the player loses connection or their game terminates for any reason, then they reconnect and queue again, they now have 2 tickets in the MM queue for the same mode. There is no deduplication happening on the matchmaker side. A single player could fill up the 1v1 queue with dead tickets and cause a poor MM experience for basically every other player queueing.
  • Simply, it gives malicious users a very low effort avenue of attack against any games using the matchmaker in a client authoritive fashion.
  • Using non-inactivity timeouts on tickets isn’t really an acceptable solution for this, since you don’t want to kick active players out of the queue just because they might be disconnected or malicious.
  • Using cloud code and a “database” has been suggested as a way to get around this, but in my opinion, this should just be something that works in the matchmaker without creating another potential point of failure by the user (user being the game developers)

Suggestions:

  • Force players to keep their MM tickets alive, at least by pinging their MM ticket every 15 seconds or so. If they have a connection outage and can’t make a single request in 15 seconds, it is fair they should be kicked out of the queue.
  • Deduplicate tickets by user ID for the same mode to [help] avoid malicious behaviour.

Backfill tickets do not allow the same userId to enter an allocation they are already on the backfill ticket for

To me this behaviour is quite counter-intuitive. If a player already exists on the backfill ticket for a particular allocation, the player can not be backfilled into that allocation. This becomes problematic in cases of:

  • The server does not properly handle the player disconnecting/leaving the game and the player remains on the backfill ticket.
  • The player never joins the match after they are allocated to it (but are on the backfill ticket). I’d like to point out that the matchmaker sample code does not handle this situation at all, and instead that player will never be able to make it into their original match via the matchmaker.
  • Lastly, it simply doesn’t offer the flexibility of allowing players to rejoin matches the relevant server thinks they should still be a part of (eg: a player disconnects, but the server still wants that player to rejoin the game, so they are left on the backfill ticket). The player existing on the backfill ticket should tell the matchmaker: “Hey, player xyz is on team red for this capture the flag match! If they queue CTF, please put them in this match!”, but instead it does the exact opposite and rejects that player from rejoining.

So, what’s actually happening in this situation? From the perspective of the matchmaker, when a player queues in this state, the matchmaker sees “the player is already in game x” and for some reason decides “I need to put them in a different game, or create a new one”. I can’t imagine it’s the intended behaviour of many multiplayer game devs that a single player be allocated into multiple matches in the same queue (or multiple matches at all).

Suggestion:

  • The matchmaker backfills a queueing player into the same allocation they are already on a backfill ticket for (or at least give the option for this behaviour).

OR, perhaps optionally

  • If the matchmaker finds that a player is already on a backfill ticket, the matchmaker scrubs the player off the existing matchmaking ticket they are a part of.

There is an odd bug with queueing for a pool with a minimum player count of 1

Not much to say about this one, this seems like a straight up bug. Steps to replicate:

  • Have a queue pool where only 1 player is required to start an allocation (should be a backfill ticket to fully replicate the conditions I’m seeing it occur under)

  • Servers should be unavailable (ie: new build that is not installed yet with forced update to new version)

  • Create a matchmaking ticket to enter that matchmaking queue, and wait.

  • If you wait for a decent period of time before the allocation is ready, you’ll instead see that the ticket got assigned to multiple allocations. I’ve seen this spin up up to 5 server allocations in response to a single matchmaking ticket.

  • [maybe relevant]: The allocated server should immediately create and add the player ID to the backfill ticket (this is what the sample code does, and may be a cause of general backfill bugs).

  • Here is an example:

  • Matchmaking ticket d2c33b27-1db0-4b7f-bae4-98158e2bf923

  • Resulted 2 match IDs being created in pool 8cb650c0-132f-4616-ba5a-ca7cb0978ed7

  • 9e2f6a4a-12f8-4689-86a0-71debebb3a6f, 94cdf5a4-7272-4c2f-807f-c8b78f3f2ca8

2

Hi ! Thank you for taking the time to provide that feedback !

Here is a couple of thoughts that I hope will answer your questions:

User is not required to keep their matchmaking ticket “alive”.

We do not find that there is a need for this.
First of all disconnection can happen at anytime and keeping alive a ticket does not guarantee that a player will join.

Second, even if with a keep alive there is no guarantee that a player that did not keep alive their ticket will not get match, same way as we cannot guarantee at 100% that when a player deletes their ticket the matchmaking is cancelled. They may have been put in a match and now are waiting for an allocation.
Finally, we do not allow to create more than one ticket per player. The response when a ticket already exists is a 429 error code, and the body contains the id of the ticket previously created. It is then up to the game developer to decide if they want to keep the ticket, or delete and recreate a new one.

Backfill tickets do not allow the same userId to enter an allocation they are already on the backfill ticket for

That is true. And that is the expected behaviour.
Matchmaker does not keep track of which match the players are currently. Therefore it cannot guarantee that a player that has not been removed from a backfill ticket will end up in the match they were in. Instead they may end up either in a new match OR in the match they previously were. This behaviour is not consistent therefore we decided to not pursue it.
Plus a player leaving a match may just really want a new match rather than being put in the same previous match.

With our current behaviour the expected result is always to put the player in a different match. In addition, a game server should always remove the team and the list of player in the match so that matchmaker can properly backfill that match.

If that is a behaviour you really need, what we can suggest though is to just save locally the IP:Port of the server the player were connected and use them to reconnect to the server

There is an odd bug with queueing for a pool with a minimum player count of 1

The behaviour you are describing is linked to the fact that when updating image on Game Server Hosting, it takes more time for a server to spin up for the first allocations request coming from matchmaker.
What happens internally in matchmaking is that after a match is made, matchmaker request san allocation to Game Server Hosting. In the process of updating an image, this allocation will take a bit more time that usual.
After a lille while, matchmaker will cancel the match and put back the ticket in matchmaking, therefore requesting a new allocation. The reason we put the ticket back in matchmaking is that is a normal matchmaking process for a live game, we try to put players as fast as possible in a match. So if an allocation takes too much time, there is a possibility that in between a request for backfill as been created that is suitable for the players.
This is the behaviour you are seeing. We will update the FAQ to describe it better.

3

Hi guillaumedUnity and thanks for the response! I will add additional thoughts based on the response:

First of all disconnection can happen at anytime and keeping alive a ticket does not guarantee that a player will join.

I do understand that a ticket being kept alive doesn’t necessarily mean the player will join, however in my opinion there’s quite a difference in allocating a match for a player that disconnected 10 minutes ago vs one that just chose not to join, or didn’t join because of an issue that happened late into the queue (like cancelling a queue as it pops). The former probably occurs at a higher frequency, especially when tickets aren’t cleaned up on game exit, for whatever reason, and could be a source of frustration for players entering dead matches.

Finally, we do not allow to create more than one ticket per player.

I really think you should look into this because this is demonstrably false, at least in my own environment. A single player can queue once, twice, a hundred times for the same match mode and those tickets will remain active until they time out. I just tested this both using a service account to create matchmaking tickets (with the same userid over and over) and using Unity clients to do the same thing.

Namely, I made a button that would execute this code:

        Task.Run(async () => {
            try
            {
                Debug.Log("queue 1v1");
                await MatchmakerService.Instance.CreateTicketAsync(players, options);
            }
            catch (Exception e)
            {
                Debug.LogException(e);
            }

        });

Now, if I hit that button 10 times on a client (as long as they are spaced apart by 1 second so they will not be 429 rate limited), that will create 10 matchmaking tickets for the 1v1 queue. I can now, on an alternate client, queue for a 1v1 and it will pop. Then I can exit the match (or just ignore the ticket popping) and queue for 1v1 again. It will pop again. And again, until the 10 tickets put into the queue by the above script have all been consumed by matches.

Therefore it cannot guarantee that a player that has not been removed from a backfill ticket will end up in the match they were in.

From my experience, it guarantees the exact opposite. If a player is on the backfill ticket for an allocation, the player will, without fail, never be placed in that match until they are removed from the backfill ticket. The moment the player is removed from the backfill ticket, they are able to re-enter the game via the matchmaker. I would love to see the behaviour of what the matchmaker does in this situation become an option: re-assign to existing match or create new match. I agree, sometimes the player may want to find another match. But there may be 9 other people in the previous match being deprived of a player.

If that is a behaviour you really need, what we can suggest though is to just save locally the IP:port of the server the player were connected and use them to reconnect to the server

I did and do still consider this an option, but as far as I’m aware, there’s no way for the player to check that the server is still online and in the same match ID as when they were allocated, so the player could enter a completely different match or just outright fail the connection. Far from unhandleable (though connecting to the wrong server would be a bit of a janky experience), but it would be nice if there was a better way of doing this.

Regarding the bug for allocations taking too long because there was recently a build, I frequently see this occur not only after a build, but just when the servers aren’t warm anymore, which is typically about 30 minutes after I last queued (which I think is the TTL causing a shutdown). It usually takes several minutes for the allocation to spin up, which also triggers this bug. Although it’s not a big deal during development, this will likely cause issues on a live game with a decent number of players where there just aren’t any available allocations because they are being used up by empty matches. It would be really nice if this wasn’t the case, or maybe we could set our own timeout on matchmaker allocations.

2 Likes