17 replies to this topic

[Nightbird]

Using the QP data provided by Scurro for the seasons 14, 15, and 16. I ran the statistical software I use for work (only 20kusd), to see if there are any interesting patterns in how personal performance affects your odds of win/loss in MWO.

I will keep the OP as simple as possible, you're welcome to ask for details or challenge my methodology in posts.

Starting with season 14, using available variables: mech weight, matches played (match), avg score per match (score), kills per match (kpm), and win percent (wpct), I started with a logistical regression to see which of these variables had the most weight in determining win/loss rates.




The odds ratio is the impact on your odds of winning by each variable in the left column, when holding all other variables constant. 1 means no systematic impact, a number >1 means your odds improve, and <1 means your odds worsen. The 95% confidence limits gives a range to the estimate, where if it includes 1, it means the variable has a statistically insignificant impact.

It's obvious from these numbers that there is only one significant predictor variable: your past win rate. This does not mean the other variables don't matter, it just means that when this past performance metric is available, no other information can improve upon this predictor. Your choice of mech weight, your average score, everything accounted for.

This is from season 14 data, what about season 15 and 16?



It turns out this model is remarkably consistent. There are no changes between seasons to speak of.

Okay... so what if we remove the past win rate percent as one of the variables, then we can better see the impact of other variables on win/loss. We'll start with season 14 again:




Within mech weight, it looks like heavier mechs consistently outperform lighter mechs, with Assault mechs being better than all other weight classes, Heavy mechs being better than Mediums and Lights, and Mediums being better than Lights.

The number of matches played in the season do not affect win rate, it may be different if I had total matches played overall.

Each 100 point increase in your average match Score will boost your win ratio by 30%, and each 1.0 kill per match extra will result in a 20% boost in your win ratio. Match Score is a complicated metric, however damage done is the largest component within the score. If we had total damage, KMDDs, assists, etc, I could investigate each component however it is fair to say from this data that taking many actions to earn match score will improve your chances of victory.

What about consistency of these predictors between seasons?



The shifts in the estimates from season 15 and 16 data for our 2 major predictors stay within the margins of error from season 14. This makes me satisfied with this model Heavier mechs also continue to out perform lighter mechs, however for season 15 the difference between Assaults and Heavy mechs was insignificant.


How to use this information:

Win rates are available as a metric that the Match Maker can use, as are Average Match Scores and Kills per match. However, the fact that these metrics are very significant (P-value <0.0001 for all) means that the MM does not use them in creating teams for QP. If it did, these variables would show a much reduced impact on your chances of winning because the MM would actively balance the skill level on each team.

The simplest way to improve upon the MM today is to add up the Win/Loss Rato for each team and switch players around until they are roughly equal. To maintain even weight, the player switching can occur within each mech weight class. Differences in mech tonnage is shown to have less of an impact as past win rates, avg score, etc.

This will result is much better, closer matches for everyone.

In order to create a formula with weights, I would need data on matchs, the players on each team, map, and final result. I don't have these, but the formula will be a function on past W/L, Score per match, and Kills per match.


How to improve upon this model:

As mentioned already, having damage done, KMDDs, etc available would help, but the two main components I would like are mech chassis and PSR tier. I do NOT need the pilot name, having this information anonymized will allow me to calculate how much EACH mech chassis impacts your win ratio, as well as gauging whether PSR and Matchmater has any impact.

I'd be able to tell you exactly how OP or UP each mech chassis is, on each map, so that the MM can create even better matches by weighing each player not only by their global average performance, but also by the mech choice and map selection.

Crazy? Nan, just math.

Edited by Nightbird, 18 November 2017 - 01:58 PM.

[mouser42]

I would love a 3rd option to pick what shirt color my team gets to wear


It would be nice to have a choice

[MischiefSC]

So much <3.

Keep in mind however that wins bump your match score by, I think, 100 pts? That's going to skew the value of match score as a predictor. What would help would be identifying the base reward you get for win or loss and subtract that from.the value. So if they had 100 matches and 50 wins, you'd pull 75 pts from their average and then adjust accordingly. At least if you want to identify the impact of match score independent of win/loss.

[Nightbird]

MischiefSC, on 18 November 2017 - 01:56 PM, said:

So much <3.

Keep in mind however that wins bump your match score by, I think, 100 pts? That's going to skew the value of match score as a predictor. What would help would be identifying the base reward you get for win or loss and subtract that from.the value. So if they had 100 matches and 50 wins, you'd pull 75 pts from their average and then adjust accordingly. At least if you want to identify the impact of match score independent of win/loss.

I've thought of a few improvements as well, running a slight fever today and called it after 3 hours doing this will update it later but I don't expect much of a change. You would expect Match Score to be a stronger predictor with a +100 points or whatever on a win, but it's still not significant. Making it weaker will make it even less so is what I guess will happen. We shall see!

[MischiefSC]

Nightbird, on 18 November 2017 - 02:11 PM, said:

I've thought of a few improvements as well, running a slight fever today and called it after 3 hours doing this will update it later but I don't expect much of a change. You would expect Match Score to be a stronger predictor with a +100 points or whatever on a win, but it's still not significant. Making it weaker will make it even less so is what I guess will happen. We shall see!

I think it will skew to near balance for kills/match score with w/l the clear indicator.

Still a bit of a double dip as kills contribute to match score but I admit I had expected score to trend higher than it did.

Kmdds would be strong I suspect as it solves for "kill stealing" but I'm not sure. I'm honestly surprised at how skewed it was. What it really indicates to me is that the primary things that drive wins are not well represented in stats.

If I used my setup for analytics at work for this ninjas would come get me. Jelly that you can play with yours.

How about if you just take the top 100 players? I'm curious - so there's a complex skillet for driving wins. Far more than damage/kills. However, the assumption based on other skill based systems is that at a high level everyone pretty much has the whole skillset, just varying in degrees. At the top 100 level are kills/score more representative of wins?

Edited by MischiefSC, 18 November 2017 - 02:30 PM.

[Nightbird]

I can run it with just the top 100 players, how would you select those 100 though? Most wins?

I can use the software because I freelance and I pay for the software/own it .

[MischiefSC]

Nightbird, on 18 November 2017 - 02:35 PM, said:

I can run it with just the top 100 players, how would you select those 100 though? Most wins?

I can use the software because I freelance and I pay for the software/own it .

Extra jelly now. I'm a slave to the system. I traded freedom and self respect for stock options. The nice thing about that is that with my self respect gone I'm surprisingly okay with it all.

I would go with 100 best w/l with over 80 matches or more.

[Tarogato]

My only problem is... I have a strong feeling that WLR is highly impacted by whether or not you play group queue.

People who play mostly group queue, with good players, will win matches more often, because team work. So it would be no surprise that past WLR is the best predictor of a match result, because they pretty much only drop as a group that works together and will win. Similar can be said for casual groups that mess around in random builds having fun.

Solo queue, however, your personal individual performance in terms of killing and damage will have much higher influence on the match result, because you're not winning matches just because you're in a group that works together and wins matches together. It comes down more to the individual pilots.




https://github.com/M...cker/issues/254
^ go upvote this, if you think solo and group stats should be segregated. It's a thing that bothered me ever since I started playing this game.

[MischiefSC]

Tarogato, on 18 November 2017 - 08:43 PM, said:

My only problem is... I have a strong feeling that WLR is highly impacted by whether or not you play group queue.

People who play mostly group queue, with good players, will win matches more often, because team work. So it would be no surprise that past WLR is the best predictor of a match result, because they pretty much only drop as a group that works together and will win. Similar can be said for casual groups that mess around in random builds having fun.

Solo queue, however, your personal individual performance in terms of killing and damage will have much higher influence on the match result, because you're not winning matches just because you're in a group that works together and wins matches together. It comes down more to the individual pilots.




https://github.com/M...cker/issues/254
^ go upvote this, if you think solo and group stats should be segregated. It's a thing that bothered me ever since I started playing this game.

I upvoted it so hard it just might have my baby.

[Khobai]

Quote

My only problem is... I have a strong feeling that WLR is highly impacted by whether or not you play group queue.

sometimes negatively

if you only play with one friend it shoves you in group queue

and you tend to lose a lot then

[Champion of Khorne Lord of Blood]

Khobai, on 19 November 2017 - 01:00 AM, said:

sometimes negatively

if you only play with one friend it shoves you in group queue

and you tend to lose a lot then

That depends, if you and your friend are both top 10% skill level players both of you being on the same team at once skews the match in your favor even more so than potential coordination of a larger group of lesser skilled players. A friend of mine are both in the top 10% based on the jarl's list and will group play together from time to time, We've 2v8ed a team and won against the odds after being matched up with a team of 10 players who put up one of the poorest fights I've ever seen. Personally I've won as last man against 7 remaining players who were in a 12 man group. In many cases we'll be the main contributing factors to a win even as the minority of the team.

While teamwork is OP, player skill does end up trumping teamwork if the collective skill of the team is still lower than the single player's. Teamwork truly only becomes unstoppable when you get a full team of 12 players who are all very high level players who, on their own, would be game changers, teamwork ends up being a powerful force multiplier.

[Davegt27]

I would run the analysis using kills

and see what comes up

when PGI does the masters challenge every week

the master walks away with a kill 9 times out of 10 so my conclusions is the best players (master) are the best because they have the ability to kill other Mechs

[Nightbird]

Kills per match is in there already?

Total kills is based on matches played, and not interesting.

Agreed, I forgot to write that I also want solo and GQ stats separated for leaderboards.

[MischiefSC]

Nightbird, on 19 November 2017 - 07:06 AM, said:

Kills per match is in there already?

Total kills is based on matches played, and not interesting.

Agreed, I forgot to write that I also want solo and GQ stats separated for leaderboards.

Like the github link Tarogato put up if you haven't already.

Kills per match... I want to see that for high level players though. Maybe sort top 100 match score with over 80 matches? That will reduce impact from people just getting carries to wins. Someone dropping in group queue with a good team and still crushing match score is deep end of the performance curve anyway.

I'm just curious how much securing the kill affects top level performance. When you're playing with the full skillset what's the metric that skews you up the most.

[Xavori]

MischiefSC, on 18 November 2017 - 02:29 PM, said:

What it really indicates to me is that the primary things that drive wins are not well represented in stats.

*cough*leaderboardisgarbage*cough*

Other than that, because he can't pull solo vs group data, no surprise at all on the outcome. Tarogato pretty much nailed that.

[Snazzy Dragon]

Nightbird, on 18 November 2017 - 01:03 PM, said:

Within mech weight, it looks like heavier mechs consistently outperform lighter mechs, with Assault mechs being better than all other weight classes, Heavy mechs being better than Mediums and Lights, and Mediums being better than Lights.

B-b-b-but l-l-lights a-are OP!

[Xavori]

Snazzy Dragon, on 19 November 2017 - 08:42 AM, said:

B-b-b-but l-l-lights a-are OP!

I've been thinking about that since reading the charts.

Lights are hard to play. You're literally one hit from being dead or so severely crippled you might as well be dead. But more than that, lights are full of bad mechs. This makes it really easy to hurt your team before you are even in the match.

Plus lights make it really easy to do dumb things like rushing off on your own, getting overly brave trying to take down what you think is a lone assault, and so on. The class is just one big trap for new and mediocre players.

Assaults, on the other hand, are pretty forgiving of one or two mistakes in playing. You also are never tempted to go rushing off by yourself since you can't really rush anywhere anyway.

Of course, I'm not tempted to build an Atlas lurm boat and totally screw up the weight stats

[Nightbird]

Tarogato, on 18 November 2017 - 08:43 PM, said:

My only problem is... I have a strong feeling that WLR is highly impacted by whether or not you play group queue.

People who play mostly group queue, with good players, will win matches more often, because team work. So it would be no surprise that past WLR is the best predictor of a match result, because they pretty much only drop as a group that works together and will win. Similar can be said for casual groups that mess around in random builds having fun.

Solo queue, however, your personal individual performance in terms of killing and damage will have much higher influence on the match result, because you're not winning matches just because you're in a group that works together and wins matches together. It comes down more to the individual pilots.




https://github.com/M...cker/issues/254
^ go upvote this, if you think solo and group stats should be segregated. It's a thing that bothered me ever since I started playing this game.

Having more predictors will make the estimates more accurate, however it doesn't make what's currently there less accurate if that makes any sense. Even if you're stomping in QP with 12 mans, your team will only win if you're getting higher match score and kills/match than your enemy. The current estimates doesn't care if you're in SQ or GQ, it is as accurate as possible for both when we don't know which portion of the data is from where.

Edited by Nightbird, 19 November 2017 - 11:16 AM.