Edited by Armored Yokai, 26 August 2019 - 08:07 AM.
7
The Last Match Maker Thread We Need
Started by Nightbird, Jun 08 2019 07:49 PM
248 replies to this topic
#201
-
-
- IS Exemplar
- 1,950 posts
Member
- LocationHouston,TX
Posted 24 August 2019 - 09:21 AM
False. We do not need Matchmaker balance. WE NEED MORE MECHPACKS!
#202
-
-
- The Hammer
- 1,245 posts
Member
- LocationPacifica Training School, waiting for BakPhar shares to rise
Posted 24 August 2019 - 05:24 PM
Nightbird, on 24 August 2019 - 09:18 AM, said:
I agree that MWO has never been friendly to getting a friend involved in the game, however, changing SQ is not the way to do it. 2-4 good players give such a huge advantage against 12 solos that it will clear the queue in no time.
Good thing we're talking about 2-player groups matched with 10 on each side, and not what you thought you read?
#203
-
-
- The God of Death
- 7,518 posts
Member
Posted 24 August 2019 - 05:39 PM
East Indy, on 24 August 2019 - 05:24 PM, said:
Good thing we're talking about 2-player groups matched with 10 on each side, and not what you thought you read?
Still no.
#204
-
-
- The 1 Percent
- 22 posts
Member
Posted 25 August 2019 - 05:37 PM
Nightbird, on 08 June 2019 - 07:49 PM, said:
This thread is an analysis on how to create a Match Maker (MM) for Quick Play Solo Queue that makes better quality matches than what we have today. Be warned that despite trying my best to explain it as simply as possible, it is still a tough read, proceed if you're interested and up to the challenge.
What makes a MM good or high quality?
Some people say matches that end in stomps (lopsided scores of 12-0, 12-1) are bad, but the truth is even when both teams have a 50/50 chance of winning at the beginning, stomps will still happen because of the snowball effect. That been said, if teams are unbalanced, for example if the chance of winning due to team imbalance is 10/90, the chance of a stomp is much higher. Therefore, I would like to define a good MM as one that makes teams with 50/50 chances of winning, as this maximizes the chances of a fair fight and at the same time minimizes the chances of a stomp.
How this thread is different from the others
Before, whenever people suggested improving the MM, it is with a method that is somewhat unscientific. This thread will substantiate with scientific methods the suggestion it contains. We will create a simulation with rules based on how we know the current MM works, create metrics on quality of matches the simulated MM makes, check that this quality corresponds to our current experiences. Then we will tweak the MM per our suggestion and rerun the simulation, seeing if the metrics on match quality has improved.
Simulation of Current MM
This is a tough section, to skip to the next section, just understand the numbers picked here are to create simulation results similar to what we see in game today.
To simulate the current MM, first I created 100 Tier 1 players with a hidden skill level that range from 200 to 1800 following a bell curve (normal distribution). The skill level is hidden because it is not directly accessible by the MM. From these 100 players, the simulated MM will randomly select 24 of which a random 12 will be assigned to each team. This matches the current MM in that there is no consideration to past performance.
To determine who wins and whether there is a stomp, I calculate the hidden skill total for each team. If the skill total for both teams is the same, the win chance is 50/50. If the skill totals are different, I estimate the win chance for Team 1 based on the difference. (tough: win chance = Cumulative Density Function of the Standard Normal based on the skill total difference / 800). I then generate a random uniform number between 0 to 100, if this random number is <= the win chance, Team 1 wins. To calculate if a stomp happens, if the difference between the random number and the win chance is >= 47.5, then a stomp occurred. This means for a balanced match, there is only a 5% chance of a stomp, but if a team has a win chance of 99%, the stomp chance increases to 52.5%.
After simulating each match, I record the results of the match and update the individual player stats.
Results of Simulation of Current MM
After running the simulation for 10,000 matches, I created some graphs to represent the quality of matches created by this simulation of the current MM. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/Dvkrq7X).
First, we have a summary of the WLR of the players based on their hidden skill level. It doesn't look bad, the best players have 2WLR, going down to 0.5 for the worst. (Some may think, hey, we see people with >3WLR in the Jarl's list! Keep in mind that with a database of 40k players, you'll see more extreme values of skills, but they represent <0.1% of the pop. We can add higher skilled players to this sim, but with their rarity it's not necessary or helpful.)
Then we look at the chance of winning based on the teams created by the MM, here it looks very bad. Only about 15% of matches have a decent win chance of 35-65. More than 50% of matches are guaranteed wins or losses 0-15, 85-100.
Finally, based on our way of calculating when stomps occur, average players experience 1/5 of matches as a stomp against them. However, lower skilled players lose to stomps 3X as often as high skilled players.
Hopefully these results are in the right ballpark per everyone's experiences. Perfection is not the goal here, but to use these results to simulate how much of an improvement we can expect from switching out the MM.
Simulation of Win-Loss Ratio (WLR) based MM
We make one change to the simulation above. Where before we picked 24 player out of 100 and tossed them randomly into 2 teams, instead we will first sort the 24 players based on their WLR, put the 1st (highest WLR) player onto team 1, the 2nd and 3rd onto team 2, etc etc, just as in a regular pick-up game between friends. For an example of a team being created, see (https://imgur.com/xEWJR5k) and please note that ties use a random tiebreaker.
What happens as a result?
Results of Simulation of WLR MM
After running the simulation for another 10,000 matches, I recreated the same graphs to represent the quality of matches. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/JEsoC5Q).
First up, the WLR ratio of players by their hidden skill level (as a reminder, hidden means unknowable by the MM). Some will question why all the WLR is not 1, the reason is because if it all become 1, the WLR MM would become blind and the more skilled players would start winning more as in the simulation of the current MM. Therefore, it is impossible for WLR for everyone to become 1.
Next, the chance of winning based on the teams created by the MM. For teams made with a win chance in the range of 35-65, instead of 15% of all matches, we now see 50% of all matches. Likewise, 'unwinnable' matches with a win chance of 0-15, 85-100 have dropped from >50% of all matches to 8%. The degree of improvement is extreme.
Lastly, the chance of stomps has dropped for everyone. It is most noticeable for lower and average skilled players where it has dropped by more than half. For the highest skilled players the change is minor. Keep in mind there is a minimum of 5% stomps even with perfectly even teams due to how we created the simulations, so it is impossible to get to 0.
Conclusion
I hope graphs make it clear we could expect a large improvement in the quality of matches made by using WLR to sort players into teams
What I Do
I analysis data from the development of various healthcare products. I program stats for a living basically. I've worked the breadth from all sorts of chemotherapy for cancer, vaccines for viruses, implants from spinal disk replacements, knees, heart, breasts, and all sorts of other stuff.
What makes a MM good or high quality?
Some people say matches that end in stomps (lopsided scores of 12-0, 12-1) are bad, but the truth is even when both teams have a 50/50 chance of winning at the beginning, stomps will still happen because of the snowball effect. That been said, if teams are unbalanced, for example if the chance of winning due to team imbalance is 10/90, the chance of a stomp is much higher. Therefore, I would like to define a good MM as one that makes teams with 50/50 chances of winning, as this maximizes the chances of a fair fight and at the same time minimizes the chances of a stomp.
How this thread is different from the others
Before, whenever people suggested improving the MM, it is with a method that is somewhat unscientific. This thread will substantiate with scientific methods the suggestion it contains. We will create a simulation with rules based on how we know the current MM works, create metrics on quality of matches the simulated MM makes, check that this quality corresponds to our current experiences. Then we will tweak the MM per our suggestion and rerun the simulation, seeing if the metrics on match quality has improved.
Simulation of Current MM
This is a tough section, to skip to the next section, just understand the numbers picked here are to create simulation results similar to what we see in game today.
To simulate the current MM, first I created 100 Tier 1 players with a hidden skill level that range from 200 to 1800 following a bell curve (normal distribution). The skill level is hidden because it is not directly accessible by the MM. From these 100 players, the simulated MM will randomly select 24 of which a random 12 will be assigned to each team. This matches the current MM in that there is no consideration to past performance.
To determine who wins and whether there is a stomp, I calculate the hidden skill total for each team. If the skill total for both teams is the same, the win chance is 50/50. If the skill totals are different, I estimate the win chance for Team 1 based on the difference. (tough: win chance = Cumulative Density Function of the Standard Normal based on the skill total difference / 800). I then generate a random uniform number between 0 to 100, if this random number is <= the win chance, Team 1 wins. To calculate if a stomp happens, if the difference between the random number and the win chance is >= 47.5, then a stomp occurred. This means for a balanced match, there is only a 5% chance of a stomp, but if a team has a win chance of 99%, the stomp chance increases to 52.5%.
After simulating each match, I record the results of the match and update the individual player stats.
Results of Simulation of Current MM
After running the simulation for 10,000 matches, I created some graphs to represent the quality of matches created by this simulation of the current MM. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/Dvkrq7X).
First, we have a summary of the WLR of the players based on their hidden skill level. It doesn't look bad, the best players have 2WLR, going down to 0.5 for the worst. (Some may think, hey, we see people with >3WLR in the Jarl's list! Keep in mind that with a database of 40k players, you'll see more extreme values of skills, but they represent <0.1% of the pop. We can add higher skilled players to this sim, but with their rarity it's not necessary or helpful.)
Then we look at the chance of winning based on the teams created by the MM, here it looks very bad. Only about 15% of matches have a decent win chance of 35-65. More than 50% of matches are guaranteed wins or losses 0-15, 85-100.
Finally, based on our way of calculating when stomps occur, average players experience 1/5 of matches as a stomp against them. However, lower skilled players lose to stomps 3X as often as high skilled players.
Hopefully these results are in the right ballpark per everyone's experiences. Perfection is not the goal here, but to use these results to simulate how much of an improvement we can expect from switching out the MM.
Simulation of Win-Loss Ratio (WLR) based MM
We make one change to the simulation above. Where before we picked 24 player out of 100 and tossed them randomly into 2 teams, instead we will first sort the 24 players based on their WLR, put the 1st (highest WLR) player onto team 1, the 2nd and 3rd onto team 2, etc etc, just as in a regular pick-up game between friends. For an example of a team being created, see (https://imgur.com/xEWJR5k) and please note that ties use a random tiebreaker.
What happens as a result?
Results of Simulation of WLR MM
After running the simulation for another 10,000 matches, I recreated the same graphs to represent the quality of matches. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/JEsoC5Q).
First up, the WLR ratio of players by their hidden skill level (as a reminder, hidden means unknowable by the MM). Some will question why all the WLR is not 1, the reason is because if it all become 1, the WLR MM would become blind and the more skilled players would start winning more as in the simulation of the current MM. Therefore, it is impossible for WLR for everyone to become 1.
Next, the chance of winning based on the teams created by the MM. For teams made with a win chance in the range of 35-65, instead of 15% of all matches, we now see 50% of all matches. Likewise, 'unwinnable' matches with a win chance of 0-15, 85-100 have dropped from >50% of all matches to 8%. The degree of improvement is extreme.
Lastly, the chance of stomps has dropped for everyone. It is most noticeable for lower and average skilled players where it has dropped by more than half. For the highest skilled players the change is minor. Keep in mind there is a minimum of 5% stomps even with perfectly even teams due to how we created the simulations, so it is impossible to get to 0.
Conclusion
I hope graphs make it clear we could expect a large improvement in the quality of matches made by using WLR to sort players into teams
What I Do
I analysis data from the development of various healthcare products. I program stats for a living basically. I've worked the breadth from all sorts of chemotherapy for cancer, vaccines for viruses, implants from spinal disk replacements, knees, heart, breasts, and all sorts of other stuff.
The6thMessenger, on 08 June 2019 - 08:59 PM, said:
If only PGI actually listens to the community.
You did a lot of work for that post NightBird, others have done so on the same subject several times, but it was a complete waste of your and the community's time just as The6thMessanger commented on. This is why we have an Epic game store exclusive for MW5, the last game PGI will produce!!!
#205
-
-
- The God of Death
- 7,518 posts
Member
Posted 26 August 2019 - 08:59 AM
Janet Yellen, on 25 August 2019 - 05:37 PM, said:
You did a lot of work for that post NightBird, others have done so on the same subject several times, but it was a complete waste of your and the community's time just as The6thMessanger commented on. This is why we have an Epic game store exclusive for MW5, the last game PGI will produce!!!
The threads I've seen on the topic were fairly subjective, didn't do metrics on improvements, etc etc.
The level of analysis here would cost 10-25k USD to buy, it's how I make my living.
#206
-
-
- The God of Death
- 7,518 posts
Member
Posted 11 September 2019 - 06:00 PM
Monthly reminder the MM can easily be improved
#207
-
-
- 216 posts
Member
- Twitter: Link
- LocationArizona, USA
Posted 11 September 2019 - 10:13 PM
MrMadguy, on 08 June 2019 - 10:28 PM, said:
I also wanted to fully simulate current MM in order to prove, that PSR is biased towards increasing. But I was too lazy to make full simulation.
Now. What is wrong with this simulation? It doesn't take many important factors into account. For example, that all 100 players aren't always available at the same time. That there is always constant flow of new and retired players. I.e., yeah, in all-vs-all situation WLR matchmaker would work due to law of large numbers. But it won't work in real conditions. That's why AvgMS part is very important. Because when you compare A + B + C with X + Y + Z, it's impossible to distinguish, what is A's contribution into A + B + C. And AvgMS allows you to do it.
That's, why I decided to focus on my own PSR level. Only thing, we don't know about it - is actual PSR change values. Unfortunately I don't have full stats from all matches, I've ever played. That's why I have to guess, that they have some normal distribution. If I would have full stats from at least one player, I would be able to fully crack PSR system then. What I currently suspect - is that for matches below 400MS it can be balanced. But any match above 400 will boost your PSR dramatically.
Now. What is wrong with this simulation? It doesn't take many important factors into account. For example, that all 100 players aren't always available at the same time. That there is always constant flow of new and retired players. I.e., yeah, in all-vs-all situation WLR matchmaker would work due to law of large numbers. But it won't work in real conditions. That's why AvgMS part is very important. Because when you compare A + B + C with X + Y + Z, it's impossible to distinguish, what is A's contribution into A + B + C. And AvgMS allows you to do it.
That's, why I decided to focus on my own PSR level. Only thing, we don't know about it - is actual PSR change values. Unfortunately I don't have full stats from all matches, I've ever played. That's why I have to guess, that they have some normal distribution. If I would have full stats from at least one player, I would be able to fully crack PSR system then. What I currently suspect - is that for matches below 400MS it can be balanced. But any match above 400 will boost your PSR dramatically.
You don't need a full simulation for that. PGI has admitted that already.
Nightbird, on 11 September 2019 - 06:00 PM, said:
Monthly reminder the MM can easily be improved
The reminder to the reminder: PGI don't give a ****.
#209
-
-
- The God of Death
- 7,518 posts
Member
Posted 13 September 2019 - 05:11 AM
JRcam4643, on 11 September 2019 - 10:13 PM, said:
The reminder to the reminder: PGI don't give a ****.
PGI does care about an empty wallet though
#210
-
-
- Bad Company
- 1,328 posts
Member
- LocationIn transit to 1 of 4 possible planets
Posted 13 September 2019 - 06:01 AM
Hmmmm played for first time last night in very long long time, MM isn't great, but it's not complete dogsh1t either, for dogsh1t match making please go play World of Warships.
Seriously though unless you have experienced the cancer that is WoWs, you have little to complain about, trust me it could be much much worse.
Seriously though unless you have experienced the cancer that is WoWs, you have little to complain about, trust me it could be much much worse.
#211
-
-
- Bad Company
- 1,273 posts
Member
Posted 20 September 2019 - 08:56 AM
Interesting analysis, could you also enrich the simulation by adding the mech class/weight distribution ? ( from Jarls or some other metric ).
It would be interesting to see if there would be great changes as team weight does make a difference as even top players don't always play assaults and heavies even though those have the potential for a greatest chance to win and some matches might turn out ridiculously weight tilted to one team if this is not taken in to account.
It would be interesting to see if there would be great changes as team weight does make a difference as even top players don't always play assaults and heavies even though those have the potential for a greatest chance to win and some matches might turn out ridiculously weight tilted to one team if this is not taken in to account.
#212
-
-
- Ace Of Spades
- 87 posts
Member
- LocationAustin
Posted 20 September 2019 - 10:37 AM
Finally! SAS used for Good!
#213
-
-
- Ace Of Spades
- 542 posts
Member
- LocationFlorida, USA
Posted 20 September 2019 - 10:48 AM
Feral Clown, on 23 August 2019 - 05:31 PM, said:
Wut?
It is obvious you never played in the NBT league.
#214
-
-
- The God of Death
- 7,518 posts
Member
Posted 20 September 2019 - 12:50 PM
Nik Reaper, on 20 September 2019 - 08:56 AM, said:
Interesting analysis, could you also enrich the simulation by adding the mech class/weight distribution ? ( from Jarls or some other metric ).
It would be interesting to see if there would be great changes as team weight does make a difference as even top players don't always play assaults and heavies even though those have the potential for a greatest chance to win and some matches might turn out ridiculously weight tilted to one team if this is not taken in to account.
It would be interesting to see if there would be great changes as team weight does make a difference as even top players don't always play assaults and heavies even though those have the potential for a greatest chance to win and some matches might turn out ridiculously weight tilted to one team if this is not taken in to account.
Not enough data to simulate that, but if PGI were doing it it would be simple to adjust the suggested MM by weight class and even chassis and mech build. Not that they would implement this anyways
OldSchoolCav, on 20 September 2019 - 10:37 AM, said:
Finally! SAS used for Good!
Ah you know the software I'm using? It's used in pharma, but also in banking if that is what you're referring to?
#215
-
-
- Big Daddy
- 717 posts
Member
Posted 04 November 2019 - 09:55 AM
I didn't read it, but there's numbers and charts and stuff. Must be good amirite?!
Whatever it is, it beats the tier system.
Whatever it is, it beats the tier system.
#217
-
-
- The God of Death
- 7,518 posts
Member
Posted 04 November 2019 - 10:33 AM
DrtyDshSoap, on 04 November 2019 - 09:55 AM, said:
I didn't read it, but there's numbers and charts and stuff. Must be good amirite?!
Whatever it is, it beats the tier system.
Whatever it is, it beats the tier system.
That's how we got our current match maker....
#218
-
-
- Private
- 543 posts
Member
Posted 05 November 2019 - 07:42 AM
Dear pgi, I would open my wallet .
Edited by Paul Meyers DEST, 05 November 2019 - 07:42 AM.
#220
-
-
- The God of Death
- 7,518 posts
Member
Posted 31 January 2020 - 02:55 PM
Matt Newman, on 30 January 2020 - 08:28 AM, said:
MWO already has a steep learning curve but an unwelcome vibe could be the thing that turns people away. Having a "good" Community is all of our responsibilities and offering new players help will go a long way to players sticking around and learning.
Yes, the community is sometimes at fault, but PGI is also at fault for creating a MM that makes bad matches 50% of the time. How about improving/reducing this to 10%?
Share the responsibility a little.
11 user(s) are reading this topic
0 members, 11 guests, 0 anonymous users
