344 replies to this topic

[Paul Inouye]

Hey MechWarriors,

As I mentioned last week, I'd be playing with a few of the match making numbers on the live servers throughout the weekend and I'd share info that was discovered in doing so.

So the initial numbers on Friday were showing me this:

Average wait in QP queue: ~48 seconds
 
Tightened Tier Separation wait time: ~75 seconds
 
Tightened Weight Class Restrictions wait time: ~59 seconds
 
Combined Tier/Weight Class Restrictions wait time: ~1 minute 30 seconds

That being said, as more tests happened, the more accurate those numbers would change to. These would be:

Average wait in QP queue: ~48 seconds
 
Tightened Tier Separation wait time: ~91 seconds
 
Tightened Weight Class Separation: ~58 seconds
 
Combined Tier/Weight Class Restrictions wait time: ~1 minute 40 seconds.

It became very clear that there was a heavy trade-off in wait times doing even minimal adjustments to the release valves and hard limits on Tier/Weight Class. Essentially it boils down to the same principle of economics:



Basically we are putting a point (set of MM values/restrictions) somewhere in the above triangle. Unless the point is dead center, there's a give and take effect happening. If that point leans toward any given corner of the triangle, the other two properties will take a hit in balance.

On Tuesday, when the queues were combined, there were very little restrictions in terms of Tier/Weight Class restrictions. It resulted in games kicking off very quickly.

Tuesday:


While a lot of feedback was positive to the combination, there were concerns regarding Tier and Weight Class disparity between teams. We investigated what could be done with the current configuration tools. The first change I made on Friday was to tighten the Tier/PSR limits. This resulted in drastic increases in wait times as would be expected:

Friday/Saturday: Tier tightening


In the above image, the Tier tightening uses the average PSR of the teams and limits how wide the MM will go before expanding to include a wider skill set of players. Even minor changes to this setting results in significant queue time increases. It also leaves the weight class issue wide open and the disparity between teams is messed up.

Friday/Saturday: Weight Class tightening


The next test above, was to see the effects of clamping Weight Class restrictions. Tiers/PSR was reset to no restriction. This Weight Class restriction resulted in tighter balancing of weight classes and increased queue wait times to a notable level. Obviously Tier/PSR balancing took a hit here.

Friday/Saturday: Tier/Weight Class combination tightening


In the image above, I tried to find a happy medium where time to get a match, Tier/PSR matching and Weight Class matching all had their place. The combination of Tier and Weight matching is additive so no matter what the settings are set to, the global wait time is going to increase. It now comes down to how much time is acceptable to get the Tier balancing and Weight balancing having a positive impact.

The current values are giving us the following:

Average wait time in QP Queue: ~1 min 30 seconds (up from ~48 seconds)
Average Tier separation between teams: ~300 (tightened from ~1800)
Average Weight Class difference: +/- 1 or 2 in a given weight class (5 heavies, 3 assaults for example) (tightened from 12)

There are times where the Match Maker runs out of players. When this happens and the queue is waiting to fill beyond release valve timers, there will be the odd circumstance where all valves are open and you will see a full open Tier/Weight Class match formed. While this is fairly rare, it is possible.

For the time being, we're going to leave the numbers where they are and continue to monitor what's going on.

[VonBruinwald]

First!

Is there an announcement for this thread? Don't think anyone will notice as is?

Are those two weight distribution charts meant to be Team 1 vs. Team 2? Both sides are labelled as Team 1.

Edited by VonBruinwald, 04 May 2020 - 04:22 PM.

[nuttyrat]

Would be interesting to see what the match quality is like.

I think this also exposes what a lot of folks were saying in other threads about PSR, that a re-work might be a good idea to further improve the quality of the matches played. Fast matches are great, but fast & good matches would be awesome!

Edited by nuttyrat, 04 May 2020 - 04:30 PM.

[Gagis]

Yay! Data! I love data!

[50 50]

Thanks for the information Paul and have to say, pretty much expect this sort of thing.
If with all the restrictions in place (weight classes and tier) it's a wait time of ~ 1 min 30 sec, that to me would be an acceptable timeframe to at least test these boundaries with.
IF these restrictions improve player experience (at both ends of the spectrum) then we could suggest that this is a positive change overall.

I do have a couple of questions:

Is the weight class restrictions for teams of the 3/3/3/3 still a viable option?
If so, is that applicable to groups as a 1/1/1/1 restriction?

Given it seems pretty easy to change the team sizes for matches, is there a possibility of making that a bit more dynamic and initially trying to create a 12v12 match, but failing that go for 8v8 or then to 4v4?

Part of this equation is the player population available to drop into a match, within those tier and weight restrictions at any given time and we may never be able to rely on that always being 24 players.
Allowing the flexibility in the system would ease that to as low as 8 players.

[Brauer]

Info on match quality would be helpful even though that can be tough to judge using just numbers like the distribution of different results (12-0, 12-1, etc.).

Would also be helpful to learn why these release valves have to be opened up so wide at this point. I had never noticed them being pulled so far open under the old solo queue system, flawed as it was.

[Paul Inouye]

VonBruinwald, on 04 May 2020 - 04:14 PM, said:

First!

Is there an announcement for this thread? Don't think anyone will notice as is?

Are those two weight distribution charts meant to be Team 1 vs. Team 2? Both sides are labelled as Team 1.

DARNIT!!! Copy and paste error in Photoshop! Yes.. those are 2 teams.

[VonBruinwald]

Gagis, on 04 May 2020 - 04:26 PM, said:

Yay! Data! I love data!

[Windscape]

Very interesting.

-balancing by fast queues mean't horrible, unbalanced, but quick match wait time.
-balancing by weight class mean't horrible, unbalanced, moderate match wait time.
-balancing by PSR meant slightly more balanced, very long match wait time.

Balancing by a single metric mean't a bonus in one area, with a trade-off in the other.
But, like I and many others said before this test went live, there would be one metric that would mean increased balance and shorter times.

REPLACING THE PSR.

- With population in mind, a more competitive Pilot Skill Rating system would allow for people to play in their respective skill level, that is, "dirty comp players" play together and "I wanna play with fren" play together. Having an actual competitive, well designed matchmaking system might allow for better long term growth as people would actually want to come back if matches are going to be more competitive.

-If population is an issue, then the solution would look something like this imo.
4 man of 90%
rest of groups/solos below 60%
other group takes the highest 2 or 3 man
then the rest of the solos that are above 90%
and then whoever's left.

Please, just replace PSR, I want to actually enjoy playing this game relatively competitively outside of community leagues.

Edited by Windscape, 04 May 2020 - 04:42 PM.

[East Indy]

Were any changes made between last night and now? Games are really one-sided like they were initially. Luck has put me on the winning side more often, but that's not the point.

[Spare Knight]

Myself, I would be willing to say that if you can keep it to 2 minutes in peak time and 5 minutes max, I would be able to live with that.

[Spare Knight]

I would even go so far as to say, 3 minutes is not too long to wait, but if it hits 5 minutes, open the valves. Hopefully, at that point, we don't have an abundance of overwhelming teams.

[Excessive Paranoia]

Paul Inouye, on 04 May 2020 - 12:52 PM, said:

Hey MechWarriors...

An average of 142.25 seconds is not a "Very Long Queue Time"... its better than what I was experiencing pre-patch in the solo queue and better by a mile for anyone who wanted to play in a group. IMO, if you really have these controls, the point on that triangle you should be aiming for is bottom-dead-center, or perhaps bottom, left of center. That would provide proper match quality, while including groups in the mix, and allowing for "decent" queue times.

Edited by Excessive Paranoia, 04 May 2020 - 04:52 PM.

[Spare Knight]

In the other thread, it seemed that everyone in the EU was happy and they were not seeing the unbalanced games that we see in prime time North America. Since prime time NA is also the busiest time for the servers, tighter match making should work. Are you guys willing to give that a shot?

[VonBruinwald]

Paul Inouye, on 04 May 2020 - 12:52 PM, said:

Average Tier separation between teams: ~300 (tightened from ~1800)

How does Tier actually work on the backend?

It seems most players (including myself) believes there's 5 tiers, and the MM simply pulls players from each tier as a whole but this number seems to imply there's a linear scale in place. e.g 0-5000 and the matchmaker is attempting to find players within range of each other regardless of Tier as it's prestented to the player.

If so, where (if it does) cap out, where do the Tier boundries lie on this scale.

Or am I speculating wildly?

[Knight Captain Morgan]

nuttyrat, on 04 May 2020 - 04:25 PM, said:

Would be interesting to see what the match quality is like.

[redacted]

[mod]for unconstructive content
Please keep the discussion constructive, folks![/mod]

Edited by GM Patience, 06 May 2020 - 12:25 PM.

[RickySpanish]

So what do those tier/psr numbers mean? They seem quite granular compared to what I was expecting, which was a simple rank from 1 to 5 or 0 to 4. Am I correct in assuming they are just overall progress to max tier? Yes I suppose that must be it. By the way, why not just give group queue back to those who want it and have that support 8v8? Seems like that would be the best idea no? Solo/limited group queue for those up for a quick game solo or with a few friends, and 8v8 for those who think they are only trolling the current system to prove how "wrong" it is

[SoulRcannon]

So there are notable wait time issues when PSR is used to match players together. Aside from (once) separating tiers 1&2 from tiers 4&5, I think we've seen in-game how PSR rating can't do much to accurately gauge relative skill levels when matching teams. I can't help but question how helpful using PSR to balance matches ultimately is at all in it's current state.

Why not give the matchmaker a better tool to use in PSR when using it to influence match/team balance? Would a tweaked zero sum PSR system improve attempts to balance the queue, and the speed of grouping teams together, by providing what makes those matches with a better metric of participant player skill? If so, then it should be considered.

By "zero sum PSR" I mean making it so a player is just as likely to lose PSR as they are to gain it between games (depending entirely on match performance). This would result in players becoming more evenly distributed between tiers over time rather than all skewing toward tier 1 (while we're at it, why not have an "S" tier reserved for those at/above 95th percentile). At that point, then you have something that has the capacity to better influence some of the issues here.

[Nearly Dead]

I have been sitting watching the weight balance box for a minute or two if necessary to avoid dropping when the weight classes are crazy out of balance. Once I click to join it only takes a few seconds. (Monday afternoon and evening EST, US server.)

Games were much better tonight, weights were not perfect but close enough. Had a drop where we had no lights, another where we had one assault but the games were closer with fewer stomps and nothing crazy. One drop we went up against a unit of 4 in Alpha plus another member of the same unit in Charlie. It was a good hard game and we won. Very enjoyable. Had some people calling matches which was great and led to some good matches.

[mr bear]

Paul,
This is across all 3 servers yes?
Any idea on stomps vs close matches based on the data collected so far? ie. Looser tier means higher stomps etc...