159 replies to this topic

[Paul Inouye]

So as I mentioned on Twitter, I stole the keys to the Match Maker configuration tool. My main purpose was to look at Solo Queue and it's current performance and the fact that Tier 1 players and Tier 4 players were being allowed in the same match right from the get go.

My main thought process was to identify the major time hogs when it came to creating the teams of individual players.

Here's what WAS happening:

1) Player skill Tier was set to 3. (Tiers are 0 based; meaning If I wanted Tier 1 and Tier 2 to be the only Tiers mixed, I'd have to set this value to 1). This value was set to 3 because of complaints of long wait times when it was reduced to 2. Theoretically, this number should be set to 0 BUT that would result in Tier 1 and Tier 2 players sitting in queue for a VERY long time.. think like 45 mins +. That being said, players capable of playing at Tier 1 to Tier 3 skill levels will allow closer competitive gameplay while maintaining shorter wait times for matches, so a value of 2 would be ideal here.

2) The Match Maker was relying on building teams using the 3/3/3/3 weight class restriction rule. We all know when looking at the weight class distribution in the current quick play queue information that this is going to be hard pressed when you have 12% of queued players playing lights, 15% in mediums, 60% in heavies and 33% in assaults. When you have queues like that in conjunction with the restraints of the above point dealing with Tiers, you can see where this is compounding to the problem.

2a) To add to the above point, the match maker would refuse to budge on 3/3/3/3 building for a minimum of 60 seconds before releasing the restriction slowly to allow a combination of up to 5 per weight class. THIS was the key to the wait times being seen in Solo QP. Once the valves opened up, teams were being created a lot faster.

So what did I do to address this? Here's the scoop.

1) Stop allowing Tier 1 players to play against Tier 4 and 5 players completely. Hard line.. no ifs ands or buts no matter how low the player count is in off peak times. To do this, the value for Tier separation has been set to 2. The biggest skill gap allowed now is Tier 1 to Tier 3. Tier 2 to Tier 4. Tier 3 to Tier 5.

2 & 2a) The Match Maker is going to wait no longer than 20 seconds to come up with a 3/3/3/3 build. After that 20 seconds is up, it will INSTANTLY allow 6/6/6/6 weight class distribution and eventually widen up to 12/12/12/12 over the next 30 seconds. The Match Maker will also weight class match on both teams. If there are 6 assaults on one team, there's guaranteed to be 6 assaults on the other. This also helps SIGNIFICANTLY when the marketing team runs an event using a single 'Mech as it's reward giving element.

Blah blah blah.. numbers number numbers... what does this mean to me? Here's the TL;DR

* Wait times have been reduced between 40-50%.
* MAXIMUM average wait times have dropped from 150 seconds to 80 seconds.
* Matches being played are a lot more challenging.
* Average PSR ranking differences between two teams has gone from 950 to 200.

I've monitored this data over the last 3 days watching all 3 prime times (NA/EU/OC) and non-prime times and the data is consistent.

Something you can do to help me:

If you think you are in a SOLO QP match that has a Tier 1 player mixed with a Tier 4 or Tier 5 player, I need that match's ID number found on the end of round screen. This way I can look up all players involved and confirm the problem and get it looked at by engineers to see where it went wrong.

Edit:Tarogato pointed out Match ID isn't available in public matches. A screenshot of the match scoreboard will help me as well. Will just take longer to look everyone up. I'll put in a request to have Match ID posted on public matches as well.

You've queue solo but you've sat there for 3 minutes!

Yes... I've seen it. I've looked into why as well. This will happen when you're playing in a non-primetime time slot while having only 1 regional server selected. i.e. You are playing at 3am in AU and have only the OC server selected. You're going to be there for quite a while. All I can suggest is for you to open up the other server selections (I know ping isn't ideal). You have a choice of waiting a long time, or dealing with higher pings and until some international Internet body allows me to play with routing information and dealing with companies that cut ocean floor pipes, there's not much else I can do for you.

So that's it for now. I'll be keeping an eye on the wait times and I've made my own copy of the keys to the server config tool.

[Tarogato]

On the topic of optimising matching quality versus queue time, have you guys ever looked at post-matchmaking team balancing?

See the very end of this OP, the "commentary" section for what I'm talking about. https://mwomercs.com...is-of-the-12-0/

There is an opportunity "swap" players between team 1 and team 2 after teams have been built to make better balanced matches in very many cases. For example, having a team with four Tier1 players whose average matchscores are 350+ is not balanced against four Tier1 players on the opposing side who average a mere 260 matchscore.

[TWIAFU]

Thank you Paul for detailed information.

/respect.

o7

[Paul Inouye]

Tarogato, on 30 May 2018 - 02:17 PM, said:

On the topic of optimising matching quality versus queue time, have you guys ever looked at post-matchmaking team balancing? See the very end of this OP, the "commentary" section for what I'm talking about. https://mwomercs.com...is-of-the-12-0/ There is an opportunity "swap" players between team 1 and team 2 after teams have been built to make better balanced matches in very many cases. For example, having a team with four Tier1 players whose average matchscores are 350+ is not balanced against four Tier1 players on the opposing side who average a mere 260 matchscore.

Yes, this is something I looked into and how it would help out the situation more. I won't comment further on it until I know IF it gets into the engineers hands and it's well under way. Until then, just mark it down as, yes... known... just don't have a delivery time frame.

[Tarogato]

Paul Inouye, on 30 May 2018 - 02:10 PM, said:

If you think you are in a SOLO QP match that has a Tier 1 player mixed with a Tier 4 or Tier 5 player, I need that match's ID number found on the end of round screen.

Us players are not provided with matchID in public queues. Best we can do is provide a screenshot so you can try to query the match db based on the involved players' names.

[Paul Inouye]

Tarogato, on 30 May 2018 - 02:33 PM, said:

Us players are not provided with matchID in public queues. Best we can do is provide a screenshot so you can try to query the match db based on the involved players' names.

This will work as well. Will also put in a request for Match ID in pub matches.

[Remover of Obstacles]

Thank you very much.

[Parashurama]

Damn fine work.

* Investigated player concern.
* Provided transparency.
* Took rational action.
* Used game forum (not Twitter) to relay information to the players.
* Improved game quality of queue wait time.
* Improved game play level of matches.
* Monitored thread and made a reply.

This will likely have a greater impact on improving Time To Kill than any of the weapon balancing that has been discussed in the last year.

Get this man a medal.

Edited by Parashurama, 30 May 2018 - 03:05 PM.

[Liveish]

Paul Inouye, on 30 May 2018 - 02:10 PM, said:

[color=orange]You've queue solo but you've sat there for 3 minutes![/color]

Yes... I've seen it. I've looked into why as well. This will happen when you're playing in a non-primetime time slot while having only 1 regional server selected. i.e. You are playing at 3am in AU and have only the OC server selected. You're going to be there for quite a while. All I can suggest is for you to open up the other server selections (I know ping isn't ideal). You have a choice of waiting a long time, or dealing with higher pings and until some international Internet body allows me to play with routing information and dealing with companies that cut ocean floor pipes, there's not much else I can do for you.

So that's it for now. I'll be keeping an eye on the wait times and I've made my own copy of the keys to the server config tool.

This also happens in AP Prime time with all 3 servers checked

Faster times should be good, I feel that a Bad T1 and a good T4 is within 5% of each other.

Edited by live1991, 30 May 2018 - 04:51 PM.

[Alilua]

Nice, hope this works out and starts to address the concern in the community.

[Nightbird]

Reads like a good set of changes!

[Virlutris]

Paul Inouye, on 30 May 2018 - 02:10 PM, said:

So as I mentioned on Twitter, I stole the keys to the Match Maker configuration tool. My main purpose was to look at Solo Queue and it's current performance and the fact that Tier 1 players and Tier 4 players were being allowed in the same match right from the get go.

My main thought process was to identify the major time hogs when it came to creating the teams of individual players.

[color=orange]Here's what WAS happening:[/color]

1) Player skill Tier was set to 3. (Tiers are 0 based; meaning If I wanted Tier 1 and Tier 2 to be the only Tiers mixed, I'd have to set this value to 1). This value was set to 3 because of complaints of long wait times when it was reduced to 2. Theoretically, this number should be set to 0 BUT that would result in Tier 1 and Tier 2 players sitting in queue for a VERY long time.. think like 45 mins +. That being said, players capable of playing at Tier 1 to Tier 3 skill levels will allow closer competitive gameplay while maintaining shorter wait times for matches, so a value of 2 would be ideal here.

2) The Match Maker was relying on building teams using the 3/3/3/3 weight class restriction rule. We all know when looking at the weight class distribution in the current quick play queue information that this is going to be hard pressed when you have 12% of queued players playing lights, 15% in mediums, 60% in heavies and 33% in assaults. When you have queues like that in conjunction with the restraints of the above point dealing with Tiers, you can see where this is compounding to the problem.

2a) To add to the above point, the match maker would refuse to budge on 3/3/3/3 building for a minimum of 60 seconds before releasing the restriction slowly to allow a combination of up to 5 per weight class. THIS was the key to the wait times being seen in Solo QP. Once the valves opened up, teams were being created a lot faster.

[color=orange]So what did I do to address this? Here's the scoop.[/color]

1) Stop allowing Tier 1 players to play against Tier 4 and 5 players completely. Hard line.. no ifs ands or buts no matter how low the player count is in off peak times. To do this, the value for Tier separation has been set to 2. The biggest skill gap allowed now is Tier 1 to Tier 3. Tier 2 to Tier 4. Tier 3 to Tier 5.

2 & 2a) The Match Maker is going to wait no longer than 20 seconds to come up with a 3/3/3/3 build. After that 20 seconds is up, it will INSTANTLY allow 6/6/6/6 weight class distribution and eventually widen up to 12/12/12/12 over the next 30 seconds. The Match Maker will also weight class match on both teams. If there are 6 assaults on one team, there's guaranteed to be 6 assaults on the other. This also helps SIGNIFICANTLY when the marketing team runs an event using a single 'Mech as it's reward giving element.

[color=orange]Blah blah blah.. numbers number numbers... what does this mean to me? Here's the TL;DR[/color]

* Wait times have been reduced between 40-50%.
* MAXIMUM average wait times have dropped from 150 seconds to 80 seconds.
* Matches being played are a lot more challenging.
* Average PSR ranking differences between two teams has gone from 950 to 200.

I've monitored this data over the last 3 days watching all 3 prime times (NA/EU/OC) and non-prime times and the data is consistent.

[color=orange]Something you can do to help me:[/color]

If you think you are in a SOLO QP match that has a Tier 1 player mixed with a Tier 4 or Tier 5 player, I need that match's ID number found on the end of round screen. This way I can look up all players involved and confirm the problem and get it looked at by engineers to see where it went wrong.

Edit:Tarogato pointed out Match ID isn't available in public matches. A screenshot of the match scoreboard will help me as well. Will just take longer to look everyone up. I'll put in a request to have Match ID posted on public matches as well.

[color=orange]You've queue solo but you've sat there for 3 minutes![/color]

Yes... I've seen it. I've looked into why as well. This will happen when you're playing in a non-primetime time slot while having only 1 regional server selected. i.e. You are playing at 3am in AU and have only the OC server selected. You're going to be there for quite a while. All I can suggest is for you to open up the other server selections (I know ping isn't ideal). You have a choice of waiting a long time, or dealing with higher pings and until some international Internet body allows me to play with routing information and dealing with companies that cut ocean floor pipes, there's not much else I can do for you.

So that's it for now. I'll be keeping an eye on the wait times and I've made my own copy of the keys to the server config tool.

GG Paul, thanks for the thoughtful work and explanation.

[TheAtomiser]

Any plans to improve wait times for the oceania group queue? Its almost impossivle to find a match so me and my friends end up playing something else.

[Liveish]

TheAtomiser, on 30 May 2018 - 04:27 PM, said:

Any plans to improve wait times for the oceania group queue? Its almost impossivle to find a match so me and my friends end up playing something else.

She dead Jim

Join the AP discord and post up when in Group and other should join

https://discord.gg/Vs7VG7p

[Appogee]

Paul Inouye, on 30 May 2018 - 02:10 PM, said:

So that's it for now. I'll be keeping an eye on the wait times and I've made my own copy of the keys to the server config tool.

Paul, I appreciate your attention to this huge issue of quality of matches.

Unfortunately, your solution is unlikely to address the problem because the matchmaker relies on PSR tiers ... and those tiers are now mostly meaningless. As you know, the PSR alogrithm is heavily weighted towards all players moving quickly up the tiers. Now there are tons of players in Tier 1 who are not 'good players' by any stretch of the imagination.

Here's the fix:

• Reset tiers each Season, equally distributing active players across the tiers, on a normal curve.
  
• Reconfigure the PSR alogrithm to weight upwards and downwards movement equally.

Only with these changes will PSR-based matchmaking actually lead to teams of approximately equal skill.

Edited by Appogee, 30 May 2018 - 05:02 PM.

[Navid A1]

Thanks for the explanation Paul.

There are two things I wanted to point out:

1- [color=#1C1C1C]PSR system in its current form is not a good metric to use for determining player skill. It needs to be replaced by something similar to average match score.[/color]
Player skill rating must be a dynamic metric. The median point of the player average match score distribution should determine the average level. This way, there won't be an upward bias towards T1 at all times. even if every single player plays at top tier comp level, using a dynamic metric ensures that the players are still going to be distributed into 5 different Tiers.
the thresholds in which Tiers are determined can then be tweaked.


2- One of the biggest problems of the current matchmaker is that it treats weight class and PSR/Tier separately. As a result a Tier 1 player is expected to have the same output in a light mech as he/she has in an assault mech.

One way to solve this is to match Tier/PSR per weight class. Here is how:

Have the matchmaker pick out 24 players based on the current rules. Then, have another algorithm check the PSR/Tier balance per weight class and shuffle as required.
This way the total team PSR/Tier will be balanced for each weight class, as well the whole team

Edited by Navid A1, 31 May 2018 - 09:36 AM.

[Jman5]

Did you measure any change in the closeness of matches after adjusting the matchmaker?

Paul Inouye, on 30 May 2018 - 02:28 PM, said:

Yes, this is something I looked into and how it would help out the situation more. I won't comment further on it until I know IF it gets into the engineers hands and it's well under way. Until then, just mark it down as, yes... known... just don't have a delivery time frame.

I think some pre-match shuffling based on say the average matchscore of your last 50-100 games would really go a long way to improving an average match.

The only pitfall I worry about is the weight class issue. If I'm in my Firestarter I'm really not going to be as effective as someone of similar skill/experience in a Madcat MKII. Yet because Light mechs get their matchscore fudged a bit with special rewards, the matchmaker could overestimate my abilities in my Firestarter. I know you can't get too granular, but I think it's worth considering if you go ahead with this much needed adjustment to the matchmaker.

Edited by Jman5, 30 May 2018 - 05:26 PM.

[Kamikaze Viking]

Appogee, on 30 May 2018 - 04:55 PM, said:

Paul, I appreciate your attention to this huge issue of quality of matches.

Unfortunately, your solution is unlikely to address the problem because the matchmaker relies on PSR tiers ... and those tiers are now mostly meaningless. As you know, the PSR alogrithm is heavily weighted towards all players moving quickly up the tiers. Now there are tons of players in Tier 1 who are not 'good players' by any stretch of the imagination.

Here's the fix:

• Reset tiers each Season, equally distributing active players across the tiers, on a normal curve.
  
• Reconfigure the PSR alogrithm to weight upwards and downwards movement equally.

Only with these changes will PSR-based matchmaking actually lead to teams of approximately equal skill.

I think Pauls fixes to the matchmaker will help a lot. But they will never fix the problem fully unless PSR is also fixed as suggested above.

Here is a great thread discussing the problem and various suggestions to fix it that are mostly variations on apogee's suggestion above. https://www.reddit.c..._response_from/

[1 21 Giggawatts]

Great to see something finally being done about this... Correct me if I am wrong - essentailly rather than reducing restrictions on tier difference you are reducing adhereance to 3,3,3,3 team composition.

Now i dont mean to seem churlish here, but people have been screaming about match quality in solo for what - approximately 2 years now? And all that needed to be done was reduce inflexibility in team comp? Wow....

Ok - if this was such a trivial thing to fix - maybe paul can also look at FW - i reckon there would be some quick wins he can get there. Might bring some people back to this game - myself included.

[1 21 Giggawatts]

Kamikaze Viking, on 30 May 2018 - 05:34 PM, said:

I think Pauls fixes to the matchmaker will help a lot. But they will never fix the problem fully unless PSR is also fixed as suggested above.

Here is a great thread discussing the problem and various suggestions to fix it that are mostly variations on apogee's suggestion above. https://www.reddit.c..._response_from/

Yep 100% agree with this - pauls initial changes will work in so far as keeping new players out of T1 matches - however unless you address the fact that there are a lot of potatoes in T1 and the problems with the tiering system then this work is mostly wasted. I think the idea of resetting the tiers every season is genius :-)