Jump to content

The Last Match Maker Thread We Need


248 replies to this topic

#201 Armored Yokai

    Member

  • PipPipPipPipPipPipPipPip
  • IS Exemplar
  • IS Exemplar
  • 1,949 posts
  • LocationHouston,TX

Posted 24 August 2019 - 09:21 AM

False. We do not need Matchmaker balance. WE NEED MORE MECHPACKS!

Edited by Armored Yokai, 26 August 2019 - 08:07 AM.


#202 East Indy

    Member

  • PipPipPipPipPipPipPipPip
  • The Hammer
  • The Hammer
  • 1,207 posts
  • LocationPacifica Training School, waiting for BakPhar shares to rise

Posted 24 August 2019 - 05:24 PM

View PostNightbird, on 24 August 2019 - 09:18 AM, said:

I agree that MWO has never been friendly to getting a friend involved in the game, however, changing SQ is not the way to do it. 2-4 good players give such a huge advantage against 12 solos that it will clear the queue in no time.

Good thing we're talking about 2-player groups matched with 10 on each side, and not what you thought you read?


#203 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 24 August 2019 - 05:39 PM

View PostEast Indy, on 24 August 2019 - 05:24 PM, said:

Good thing we're talking about 2-player groups matched with 10 on each side, and not what you thought you read?


Still no.

#204 Janet Yellen

    Member

  • PipPip
  • The 1 Percent
  • The 1 Percent
  • 22 posts

Posted 25 August 2019 - 05:37 PM

View PostNightbird, on 08 June 2019 - 07:49 PM, said:

This thread is an analysis on how to create a Match Maker (MM) for Quick Play Solo Queue that makes better quality matches than what we have today. Be warned that despite trying my best to explain it as simply as possible, it is still a tough read, proceed if you're interested and up to the challenge.

What makes a MM good or high quality?

Some people say matches that end in stomps (lopsided scores of 12-0, 12-1) are bad, but the truth is even when both teams have a 50/50 chance of winning at the beginning, stomps will still happen because of the snowball effect. That been said, if teams are unbalanced, for example if the chance of winning due to team imbalance is 10/90, the chance of a stomp is much higher. Therefore, I would like to define a good MM as one that makes teams with 50/50 chances of winning, as this maximizes the chances of a fair fight and at the same time minimizes the chances of a stomp.

How this thread is different from the others

Before, whenever people suggested improving the MM, it is with a method that is somewhat unscientific. This thread will substantiate with scientific methods the suggestion it contains. We will create a simulation with rules based on how we know the current MM works, create metrics on quality of matches the simulated MM makes, check that this quality corresponds to our current experiences. Then we will tweak the MM per our suggestion and rerun the simulation, seeing if the metrics on match quality has improved.

Simulation of Current MM

This is a tough section, to skip to the next section, just understand the numbers picked here are to create simulation results similar to what we see in game today.

To simulate the current MM, first I created 100 Tier 1 players with a hidden skill level that range from 200 to 1800 following a bell curve (normal distribution). The skill level is hidden because it is not directly accessible by the MM. From these 100 players, the simulated MM will randomly select 24 of which a random 12 will be assigned to each team. This matches the current MM in that there is no consideration to past performance.

To determine who wins and whether there is a stomp, I calculate the hidden skill total for each team. If the skill total for both teams is the same, the win chance is 50/50. If the skill totals are different, I estimate the win chance for Team 1 based on the difference. (tough: win chance = Cumulative Density Function of the Standard Normal based on the skill total difference / 800). I then generate a random uniform number between 0 to 100, if this random number is <= the win chance, Team 1 wins. To calculate if a stomp happens, if the difference between the random number and the win chance is >= 47.5, then a stomp occurred. This means for a balanced match, there is only a 5% chance of a stomp, but if a team has a win chance of 99%, the stomp chance increases to 52.5%.

Posted Image

After simulating each match, I record the results of the match and update the individual player stats.

Posted Image

Results of Simulation of Current MM

After running the simulation for 10,000 matches, I created some graphs to represent the quality of matches created by this simulation of the current MM. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/Dvkrq7X).

First, we have a summary of the WLR of the players based on their hidden skill level. It doesn't look bad, the best players have 2WLR, going down to 0.5 for the worst. (Some may think, hey, we see people with >3WLR in the Jarl's list! Keep in mind that with a database of 40k players, you'll see more extreme values of skills, but they represent <0.1% of the pop. We can add higher skilled players to this sim, but with their rarity it's not necessary or helpful.)

Posted Image

Then we look at the chance of winning based on the teams created by the MM, here it looks very bad. Only about 15% of matches have a decent win chance of 35-65. More than 50% of matches are guaranteed wins or losses 0-15, 85-100.

Posted Image

Finally, based on our way of calculating when stomps occur, average players experience 1/5 of matches as a stomp against them. However, lower skilled players lose to stomps 3X as often as high skilled players.

Posted Image

Hopefully these results are in the right ballpark per everyone's experiences. Perfection is not the goal here, but to use these results to simulate how much of an improvement we can expect from switching out the MM.

Simulation of Win-Loss Ratio (WLR) based MM

We make one change to the simulation above. Where before we picked 24 player out of 100 and tossed them randomly into 2 teams, instead we will first sort the 24 players based on their WLR, put the 1st (highest WLR) player onto team 1, the 2nd and 3rd onto team 2, etc etc, just as in a regular pick-up game between friends. For an example of a team being created, see (https://imgur.com/xEWJR5k) and please note that ties use a random tiebreaker.

What happens as a result?

Results of Simulation of WLR MM

After running the simulation for another 10,000 matches, I recreated the same graphs to represent the quality of matches. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/JEsoC5Q).

First up, the WLR ratio of players by their hidden skill level (as a reminder, hidden means unknowable by the MM). Some will question why all the WLR is not 1, the reason is because if it all become 1, the WLR MM would become blind and the more skilled players would start winning more as in the simulation of the current MM. Therefore, it is impossible for WLR for everyone to become 1.

Posted Image

Next, the chance of winning based on the teams created by the MM. For teams made with a win chance in the range of 35-65, instead of 15% of all matches, we now see 50% of all matches. Likewise, 'unwinnable' matches with a win chance of 0-15, 85-100 have dropped from >50% of all matches to 8%. The degree of improvement is extreme.

Posted Image

Lastly, the chance of stomps has dropped for everyone. It is most noticeable for lower and average skilled players where it has dropped by more than half. For the highest skilled players the change is minor. Keep in mind there is a minimum of 5% stomps even with perfectly even teams due to how we created the simulations, so it is impossible to get to 0.

Posted Image

Conclusion
I hope graphs make it clear we could expect a large improvement in the quality of matches made by using WLR to sort players into teams

What I Do
I analysis data from the development of various healthcare products. I program stats for a living basically. I've worked the breadth from all sorts of chemotherapy for cancer, vaccines for viruses, implants from spinal disk replacements, knees, heart, breasts, and all sorts of other stuff.

View PostThe6thMessenger, on 08 June 2019 - 08:59 PM, said:

If only PGI actually listens to the community.


You did a lot of work for that post NightBird, others have done so on the same subject several times, but it was a complete waste of your and the community's time just as The6thMessanger commented on. This is why we have an Epic game store exclusive for MW5, the last game PGI will produce!!!

#205 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 26 August 2019 - 08:59 AM

View PostJanet Yellen, on 25 August 2019 - 05:37 PM, said:

You did a lot of work for that post NightBird, others have done so on the same subject several times, but it was a complete waste of your and the community's time just as The6thMessanger commented on. This is why we have an Epic game store exclusive for MW5, the last game PGI will produce!!!


The threads I've seen on the topic were fairly subjective, didn't do metrics on improvements, etc etc.

The level of analysis here would cost 10-25k USD to buy, it's how I make my living.

#206 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 11 September 2019 - 06:00 PM

Monthly reminder the MM can easily be improved

#207 JRcam4643

    Member

  • PipPipPipPipPipPip
  • 216 posts
  • Twitter: Link
  • LocationArizona, USA

Posted 11 September 2019 - 10:13 PM

View PostMrMadguy, on 08 June 2019 - 10:28 PM, said:

I also wanted to fully simulate current MM in order to prove, that PSR is biased towards increasing. But I was too lazy to make full simulation.

Now. What is wrong with this simulation? It doesn't take many important factors into account. For example, that all 100 players aren't always available at the same time. That there is always constant flow of new and retired players. I.e., yeah, in all-vs-all situation WLR matchmaker would work due to law of large numbers. But it won't work in real conditions. That's why AvgMS part is very important. Because when you compare A + B + C with X + Y + Z, it's impossible to distinguish, what is A's contribution into A + B + C. And AvgMS allows you to do it.

That's, why I decided to focus on my own PSR level. Only thing, we don't know about it - is actual PSR change values. Unfortunately I don't have full stats from all matches, I've ever played. That's why I have to guess, that they have some normal distribution. If I would have full stats from at least one player, I would be able to fully crack PSR system then. What I currently suspect - is that for matches below 400MS it can be balanced. But any match above 400 will boost your PSR dramatically.


You don't need a full simulation for that. PGI has admitted that already.

View PostNightbird, on 11 September 2019 - 06:00 PM, said:

Monthly reminder the MM can easily be improved


The reminder to the reminder: PGI don't give a ****.

#208 JRcam4643

    Member

  • PipPipPipPipPipPip
  • 216 posts
  • Twitter: Link
  • LocationArizona, USA

Posted 11 September 2019 - 10:26 PM

View PostArmored Yokai, on 24 August 2019 - 09:21 AM, said:

False. We do not need Matchmaker balance. WE NEED MORE MECHPACKS!


Amen brotha.

#209 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 13 September 2019 - 05:11 AM

View PostJRcam4643, on 11 September 2019 - 10:13 PM, said:

The reminder to the reminder: PGI don't give a ****.


PGI does care about an empty wallet though

#210 KHETTI

    Member

  • PipPipPipPipPipPipPipPip
  • Bad Company
  • Bad Company
  • 1,328 posts
  • LocationIn transit to 1 of 4 possible planets

Posted 13 September 2019 - 06:01 AM

Hmmmm played for first time last night in very long long time, MM isn't great, but it's not complete dogsh1t either, for dogsh1t match making please go play World of Warships.
Seriously though unless you have experienced the cancer that is WoWs, you have little to complain about, trust me it could be much much worse.

#211 Nik Reaper

    Member

  • PipPipPipPipPipPipPipPip
  • Bad Company
  • Bad Company
  • 1,273 posts

Posted 20 September 2019 - 08:56 AM

Interesting analysis, could you also enrich the simulation by adding the mech class/weight distribution ? ( from Jarls or some other metric ).

It would be interesting to see if there would be great changes as team weight does make a difference as even top players don't always play assaults and heavies even though those have the potential for a greatest chance to win and some matches might turn out ridiculously weight tilted to one team if this is not taken in to account.

#212 OldSchoolCav

    Member

  • PipPipPip
  • Ace Of Spades
  • 87 posts
  • LocationAustin

Posted 20 September 2019 - 10:37 AM

Finally! SAS used for Good!

#213 Aidan

    Member

  • PipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 542 posts
  • LocationFlorida, USA

Posted 20 September 2019 - 10:48 AM

View PostFeral Clown, on 23 August 2019 - 05:31 PM, said:


Wut?


It is obvious you never played in the NBT league.

#214 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 20 September 2019 - 12:50 PM

View PostNik Reaper, on 20 September 2019 - 08:56 AM, said:

Interesting analysis, could you also enrich the simulation by adding the mech class/weight distribution ? ( from Jarls or some other metric ).

It would be interesting to see if there would be great changes as team weight does make a difference as even top players don't always play assaults and heavies even though those have the potential for a greatest chance to win and some matches might turn out ridiculously weight tilted to one team if this is not taken in to account.


Not enough data to simulate that, but if PGI were doing it it would be simple to adjust the suggested MM by weight class and even chassis and mech build. Not that they would implement this anyways

View PostOldSchoolCav, on 20 September 2019 - 10:37 AM, said:

Finally! SAS used for Good!


Ah you know the software I'm using? It's used in pharma, but also in banking if that is what you're referring to?

#215 MTier Slayed Up

    Member

  • PipPipPipPipPipPipPip
  • Big Daddy
  • Big Daddy
  • 717 posts

Posted 04 November 2019 - 09:55 AM

I didn't read it, but there's numbers and charts and stuff. Must be good amirite?!

Whatever it is, it beats the tier system.

#216 thievingmagpi

    Member

  • PipPipPipPipPipPipPipPip
  • 1,577 posts

Posted 04 November 2019 - 10:06 AM

big thumbs up to Nightbird

#217 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 04 November 2019 - 10:33 AM

View PostDrtyDshSoap, on 04 November 2019 - 09:55 AM, said:

I didn't read it, but there's numbers and charts and stuff. Must be good amirite?!

Whatever it is, it beats the tier system.


That's how we got our current match maker....

#218 Paul Meyers DEST

    Member

  • PipPipPipPipPipPipPip
  • Bridesmaid
  • Bridesmaid
  • 539 posts

Posted 05 November 2019 - 07:42 AM

Dear pgi, I would open my wallet .

Edited by Paul Meyers DEST, 05 November 2019 - 07:42 AM.


#219 Dimento Graven

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • Guillotine
  • Guillotine
  • 6,208 posts

Posted 05 November 2019 - 09:50 AM

Bah, never mind, I thought this was a recent post, and it wasn't... Stupid thing refreshed my cookies or something.

Edited by Dimento Graven, 05 November 2019 - 09:54 AM.


#220 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 31 January 2020 - 02:55 PM

View PostMatt Newman, on 30 January 2020 - 08:28 AM, said:

MWO already has a steep learning curve but an unwelcome vibe could be the thing that turns people away. Having a "good" Community is all of our responsibilities and offering new players help will go a long way to players sticking around and learning.


Yes, the community is sometimes at fault, but PGI is also at fault for creating a MM that makes bad matches 50% of the time. How about improving/reducing this to 10%?

Share the responsibility a little.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users