248 replies to this topic

[Nightbird]

MischiefSC, on 17 August 2019 - 08:49 AM, said:

I get the logic of what you're trying to do but bell curves for human skill don't work super well. It helps, sure, but it doesn't work super well.

This is my wheelhouse; it's what I do in analytics, analyze both employee and customer performance and associated behaviors. Ironically that's where I got into gaming matchmakers because the challenges are extremely similar.

Think of what drives win/loss as being a 200 piece jigsaw puzzle. Most people only get like 50 pieces, ever. As someone collects more pieces he can make more of the puzzle. The best players may have 150 pieces -

The problem is that the vast majority of people have the same 50 pieces. 100 players each with a 1.1 W/L probably all have almost the exact same strengths and weaknesses, with tiny variation. The idea that you can build a low skill team with a couple high skill players to make a team that matches a bunch of mediocre players isn't really accurate because there's no guarantee the skill deficiencies in the lowbies are offset by that good players strengths.

What you're talking about really falls under a StrengthFinders umbrella; trying to slice performance up into specific strengths and weaknesses to then build teams toward a set performance goal based on stacking strengths and weaknesses. It's possible but hugely time intensive and in a setting like this not that reliable.

Your matchmaker idea is still great. I just want to be clear that any matchmaker for this sort of environment is going to have outliers, because humans and everyone needs to still play matches even when a good matchup isn't available.

However your idea is still going to get us better results than we have currently, and likely by a good margin. Just need to set the expectation that both ends of the curve are going to be skewed.

The more 'jigsaw pieces' that feed into a metric, actually the normally distributed (bell curve shaped) the metric will be. In stats this phenomenon is described by the Central Limit Theorem.

For example, for human height, what goes into it is nutrition and genetics, but nutrition can be broken down into calcium, vitamin A, B1, etc... a hundred components, and genetics can be broken down into gene 1, gene 2 etc, etc. As a result, human height is bell shaped. Same is true for IQ, test scores, and yes... skill

I see what you're trying to do, but you're also trying to reinvent a wheel that took many people a hundred years to perfect.

Edited by Nightbird, 17 August 2019 - 10:51 AM.

[Mystere]

Kiiyor, on 08 June 2019 - 11:36 PM, said:

What were we talking about again?

The need for random matchmaking.

[Mystere]

Nightbird, on 17 August 2019 - 09:35 AM, said:

For example, for human height, what goes into it is nutrition and genetics, but nutrition can be broken down into calcium, vitamin A, B1, etc... a hundred components, and genetics can be broken down into gene 1, gene 2 etc, etc.

There is a reason why the concept of "feature extraction" was created.

[Nightbird]

Mystere, on 17 August 2019 - 01:43 PM, said:

There is a reason why the concept of "feature extraction" was created.

Removing redundant info? For example, avgMS is redundant when you have WLR, I went over this on the first page.

In stats, it's much older concept called collinearity.

Edited by Nightbird, 17 August 2019 - 02:36 PM.

[Crockdaddy]

Nightbird, on 08 June 2019 - 07:49 PM, said:

This thread is an analysis on how to create a Match Maker (MM) for Quick Play Solo Queue that makes better quality matches than what we have today. Be warned that despite trying my best to explain it as simply as possible, it is still a tough read, proceed if you're interested and up to the challenge.

What makes a MM good or high quality?

Some people say matches that end in stomps (lopsided scores of 12-0, 12-1) are bad, but the truth is even when both teams have a 50/50 chance of winning at the beginning, stomps will still happen because of the snowball effect. That been said, if teams are unbalanced, for example if the chance of winning due to team imbalance is 10/90, the chance of a stomp is much higher. Therefore, I would like to define a good MM as one that makes teams with 50/50 chances of winning, as this maximizes the chances of a fair fight and at the same time minimizes the chances of a stomp.

How this thread is different from the others

Before, whenever people suggested improving the MM, it is with a method that is somewhat unscientific. This thread will substantiate with scientific methods the suggestion it contains. We will create a simulation with rules based on how we know the current MM works, create metrics on quality of matches the simulated MM makes, check that this quality corresponds to our current experiences. Then we will tweak the MM per our suggestion and rerun the simulation, seeing if the metrics on match quality has improved.

Simulation of Current MM

This is a tough section, to skip to the next section, just understand the numbers picked here are to create simulation results similar to what we see in game today.

To simulate the current MM, first I created 100 Tier 1 players with a hidden skill level that range from 200 to 1800 following a bell curve (normal distribution). The skill level is hidden because it is not directly accessible by the MM. From these 100 players, the simulated MM will randomly select 24 of which a random 12 will be assigned to each team. This matches the current MM in that there is no consideration to past performance.

To determine who wins and whether there is a stomp, I calculate the hidden skill total for each team. If the skill total for both teams is the same, the win chance is 50/50. If the skill totals are different, I estimate the win chance for Team 1 based on the difference. (tough: win chance = Cumulative Density Function of the Standard Normal based on the skill total difference / 800). I then generate a random uniform number between 0 to 100, if this random number is <= the win chance, Team 1 wins. To calculate if a stomp happens, if the difference between the random number and the win chance is >= 47.5, then a stomp occurred. This means for a balanced match, there is only a 5% chance of a stomp, but if a team has a win chance of 99%, the stomp chance increases to 52.5%.



After simulating each match, I record the results of the match and update the individual player stats.



Results of Simulation of Current MM

After running the simulation for 10,000 matches, I created some graphs to represent the quality of matches created by this simulation of the current MM. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/Dvkrq7X).

First, we have a summary of the WLR of the players based on their hidden skill level. It doesn't look bad, the best players have 2WLR, going down to 0.5 for the worst. (Some may think, hey, we see people with >3WLR in the Jarl's list! Keep in mind that with a database of 40k players, you'll see more extreme values of skills, but they represent <0.1% of the pop. We can add higher skilled players to this sim, but with their rarity it's not necessary or helpful.)



Then we look at the chance of winning based on the teams created by the MM, here it looks very bad. Only about 15% of matches have a decent win chance of 35-65. More than 50% of matches are guaranteed wins or losses 0-15, 85-100.



Finally, based on our way of calculating when stomps occur, average players experience 1/5 of matches as a stomp against them. However, lower skilled players lose to stomps 3X as often as high skilled players.



Hopefully these results are in the right ballpark per everyone's experiences. Perfection is not the goal here, but to use these results to simulate how much of an improvement we can expect from switching out the MM.

Simulation of Win-Loss Ratio (WLR) based MM

We make one change to the simulation above. Where before we picked 24 player out of 100 and tossed them randomly into 2 teams, instead we will first sort the 24 players based on their WLR, put the 1st (highest WLR) player onto team 1, the 2nd and 3rd onto team 2, etc etc, just as in a regular pick-up game between friends. For an example of a team being created, see (https://imgur.com/xEWJR5k) and please note that ties use a random tiebreaker.

What happens as a result?

Results of Simulation of WLR MM

After running the simulation for another 10,000 matches, I recreated the same graphs to represent the quality of matches. If you are interested, you can see the individual stats of all 100 players here (https://imgur.com/JEsoC5Q).

First up, the WLR ratio of players by their hidden skill level (as a reminder, hidden means unknowable by the MM). Some will question why all the WLR is not 1, the reason is because if it all become 1, the WLR MM would become blind and the more skilled players would start winning more as in the simulation of the current MM. Therefore, it is impossible for WLR for everyone to become 1.



Next, the chance of winning based on the teams created by the MM. For teams made with a win chance in the range of 35-65, instead of 15% of all matches, we now see 50% of all matches. Likewise, 'unwinnable' matches with a win chance of 0-15, 85-100 have dropped from >50% of all matches to 8%. The degree of improvement is extreme.



Lastly, the chance of stomps has dropped for everyone. It is most noticeable for lower and average skilled players where it has dropped by more than half. For the highest skilled players the change is minor. Keep in mind there is a minimum of 5% stomps even with perfectly even teams due to how we created the simulations, so it is impossible to get to 0.



Conclusion
I hope graphs make it clear we could expect a large improvement in the quality of matches made by using WLR to sort players into teams

What I Do
I analysis data from the development of various healthcare products. I program stats for a living basically. I've worked the breadth from all sorts of chemotherapy for cancer, vaccines for viruses, implants from spinal disk replacements, knees, heart, breasts, and all sorts of other stuff.

Just reading this now Nightbird, nice! Great write up.

[Aidan]

Sheesh there is No ONE playing this game anymore. Who gives a FF !!!

Play some Apex Legends !!!

Edited by Aidan, 20 August 2019 - 02:23 PM.

[Nightbird]

You apparently...

Aidan, on 20 August 2019 - 02:23 PM, said:

Who gives a FF !!!

[FRAGTAST1C]

Aidan, on 20 August 2019 - 02:23 PM, said:

Play some Apex Legends !!!

Is that the new "Fortnite" and "PUBG"?

[Alloh]

For me, the solution now requires also to MERGE GROUP AND SOLO QUEUES... then apply the "sort/split" method described. Since now we have smaller population, became required to have fewer sub-groups by TIER and SoloXgroup. Indeed, maybe ONE or TWO queues at most, in the second case, separating players by "recent performance" instead of "long-term cummulative tier".


For SORT/SPLIT, just treat each group as the "top" player for group... OR add a "same level" pug player for each member of group. Not sure which would provide better results, better to try both ways.

BUT more important than TIER, is the "SKILL", much better represented by RECENT W/L and K/D RATIOS than long-term sum of results. That would be more important for MM to consider.

As wild example, you can have a new account, ALT, from a top-tier player, against an old account of a bad player. The new account would be TIER5, but great WL+KD ratios, while the old bad player can be TIER1 but with quite low WL+KD ratios.

Basically, remove the "cummulative rank" and replace with a "Last Weeks performance rank".

Also refer to the same problem resulting in Units disbanding, players migrating for other games that foster teamwork and group collaboration, instead of packing up 12 random people...
https://mwomercs.com...ts-to-end-game/

Edited by Alloh, 22 August 2019 - 09:27 AM.

[Prototelis]

We've had groups in SQ. It was ******* terrible for reasons that have been outlined 1000 times.

[MadcatX]

Prototelis, on 22 August 2019 - 11:40 AM, said:

We've had groups in SQ. It was ******* terrible for reasons that have been outlined 1000 times.

I played much more often back when they had tried that out. I remember the forumwarrior outrage and anecdotal stories that were repeated 1000 times.

[Prototelis]

lol, sorry you don't understand that letting teams into solo play is bad or that limiting team size in an attempt to balance teams in solo mode is even worse.

[Nightbird]

Yeah, groups with solos will only create worse games and a more rapid decline in the player base.

I suggest groups getting some FP training and enjoying the real game, however under-developed it is. Knowing the map and mode, then picking your drop deck is what MW is about. Random maps and modes was only intended to be an extended tutorial/practice mode.

Edited by Nightbird, 23 August 2019 - 08:12 AM.

[Aidan]

Nightbird, on 23 August 2019 - 08:11 AM, said:

Yeah, groups with solos will only create worse games and a more rapid decline in the player base.

I suggest groups getting some FP training and enjoying the real game, however under-developed it is. Knowing the map and mode, then picking your drop deck is what MW is about. Random maps and modes was only intended to be an extended tutorial/practice mode.

Good suggestion ! But looking back on MechWarrior of many years ago, the term " drop deck" was not used. The term was "Drop Dec" where Dec stood for "Declaration". Before a battle was fought the map and the number and types of Battlemechs was declared by each team ! Each team was held to maintain this declaration and if when the drop started it was discovered a team did not conform to their declaration, they would forfeit the drop.

But alas, this was when leagues where bound by many rules. The Net BattleTech league or NBT, as it was known, had an amazing set of rules requiring strong leadership and many members to create a vibrant and wonderful universe of battle.

Edited by Aidan, 23 August 2019 - 12:22 PM.

[Feral Clown]

Aidan, on 23 August 2019 - 12:21 PM, said:

Good suggestion ! But looking back on MechWarrior of many years ago, the term " drop deck" was not used. The term was "Drop Dec" where Dec stood for "Declaration". Before a battle was fought the map and the number and types of Battlemechs was declared by each team ! Each team was held to maintain this declaration and if when the drop started it was discovered a team did not conform to their declaration, they would forfeit the drop.

But alas, this was when leagues where bound by many rules. The Net BattleTech league or NBT, as it was known, had an amazing set of rules requiring strong leadership and many members to create a vibrant and wonderful universe of battle.

Wut?

[MadcatX]

Prototelis, on 22 August 2019 - 03:53 PM, said:

lol, sorry you don't understand that letting teams into solo play is bad or that limiting team size in an attempt to balance teams in solo mode is even worse.

You're right, I don't understand, since I've moved on and played many pvp games over the years that allowed groups on either side in a casual queue. Even Depth, a game with an asymmetrical multiplayer structure, you could still group up and choose to be in a group of 4 on one side while the 2 on the other are solo's.

And I'll admit I've got some major bias for allowing tiny teams into solo queue. There's been too many times over the years that I've tried to introduce friends to this game, we'd group up, they'd get their butts handed to them because they were new and put into GQ. At which point the only advice I could give is to drop in the solo queue to learn the basics (Most of this is before the academy) first. All but one friend didn't stick around, they thought it was a bit silly and a horrible new player experience.

I won't deny Nightbird's great analytics. But nor will I let people invalidate my point of view, albeit subjective, I suspect I'm not the only person who ran across this issue.

And tbh even talking about balance in a game that is having longer and longer wait times and has clearly past the point of no return in terms of the inevitable death spiral along with game devs straight out saying (or with PGI, saying "we might look into that", which usually translates to "No, we're not allocating resources into looking into that) that they're done with MWO, is kind of a waste of time.

[Feral Clown]

MadcatX, on 23 August 2019 - 06:55 PM, said:

You're right, I don't understand, since I've moved on and played many pvp games over the years that allowed groups on either side in a casual queue. Even Depth, a game with an asymmetrical multiplayer structure, you could still group up and choose to be in a group of 4 on one side while the 2 on the other are solo's.

And I'll admit I've got some major bias for allowing tiny teams into solo queue. There's been too many times over the years that I've tried to introduce friends to this game, we'd group up, they'd get their butts handed to them because they were new and put into GQ. At which point the only advice I could give is to drop in the solo queue to learn the basics (Most of this is before the academy) first. All but one friend didn't stick around, they thought it was a bit silly and a horrible new player experience.

I won't deny Nightbird's great analytics. But nor will I let people invalidate my point of view, albeit subjective, I suspect I'm not the only person who ran across this issue.

And tbh even talking about balance in a game that is having longer and longer wait times and has clearly past the point of no return in terms of the inevitable death spiral along with game devs straight out saying (or with PGI, saying "we might look into that", which usually translates to "No, we're not allocating resources into looking into that) that they're done with MWO, is kind of a waste of time.

I can only surmise that the games you are referring to have a much more robust elo system or something else going on to allow for groups against solo's.

This game has none of that and letting groups into solo queue would be very, very bad. Not for me, go ahead and push for it if you like, but it will really suck for average players with average or worse friends.

[Sjorpha]

Don't put groups in the solo queue.

I agree that it's very sad if the population can no longer support an active group queue for quickplay but it's not worth ruining solo q just to allow grouping in quickplay.

[East Indy]

MWO never attempted a buddy-system insertion in solo queue.

Initially, matchmaker was matchmaker. Groups of 2, 3 and 4 could be combined with other groups, and potentially face off with teams comprised entirely of solo players. In early 2014, teams were limited to one group of 2-4 players but group sizes themselves were never enforced to be the same before Solo Queue eventually went all-solo.

We've seen sync dropping and it doesn't necessarily throw matches. The simplest test would be a limited-time functionality allowing 2-player groups to join 10 solo players — experience it without committing. Looking back, maybe one part of the week or even month could be designated as "bring-a-friend" in Solo, allowing new players to explore the game with an experienced one.

Ultimately, the elephant in the room was maintaining Group Queue parallel to Community Warfare/Faction Play. Group was the venue for off-brand guilds to push around mixed teams instead of going to the main event of FP, splitting player segments in the process and perpetuating the experience of enjoying solo play enough to show your friend how irritating group play could be.

[Nightbird]

East Indy, on 24 August 2019 - 06:52 AM, said:

I agree that MWO has never been friendly to getting a friend involved in the game, however, changing SQ is not the way to do it. 2-4 good players give such a huge advantage against 12 solos that it will clear the queue in no time.

Luckily MW5 is around the corner, you can get your friend used to shooting T5 pugs without fear of stomps.