187 replies to this topic

[Tarogato]

Ever wonder what is the driving force behind those really lopsided matches? Maybe one side had a LOT more tonnage, or just had more meta mechs, maybe it was map imbalance, or maybe matchmaker just dropped the ball and one side had better players. Maybe it was teamwork or leadership. I decided to analyse some of these stomp-matches and see if I could find anything that is statistically significant.



METHOD:

- I included only matches that ended in 12-0, 12-1, or even 12-2 if it felt particularly stomp-y
- Every match is from solo queue
- Every screenshot was submitted by a Tier 1 player
- 76% were my own matches, the rest submitted by unit mates
- Matches were collected between October 1 and January 1

Useless fact: out of the 88 matches I played, I won 67 and only lost 21. gg me

- I collected all pilot names, so that I could pull their stats from the QP Leaderboard
- I compared winning teams vs. losing teams according the average of their players leaderboard stats
- I collected all mech names, so that I could compare tonnages and mech ratings

Here is a copy of my spreadsheet with pilot names removed, but I'll explain all my findings below, I don't expect anybody to understand this poorly organised mess: https://docs.google....t#gid=288291280

(note, some variables/analysis may be missing, as I made those on my original after creating this dummy-duplicate.)





THE FINDINGS:


Total matches collected: 116
Total unique players: 2,337
Total unique chassis: 73
Total unique variants: 251


Average match score:
This is each pilot's average match score on the QP Leaderboard, not how they actually performed inside the match itself.

- Winning team average: 239
- Losing team average: 225
6.6% advantage to the teams that won.

- The team that won had the advantage 88 times (average of a 10% advantage)
- The team that lost had the advantage 28 times (average of a 3% advantage)



Win/Loss Ratio:
This is each pilot's cumulative WLR on the QP Leaderboard

- Winning team average: 1.16
- Losing team average: 1.06
9.4% advantage to the teams that won.

- The team that won had the advantage 93 times (average of a 15% advantage)
- The team that lost had the advantage 23 times (average of a 7% advantage)



Kill rate:
This is each pilot's cumulative kills per match (not KDR!) on the QP Leaderboard

- Winning team average: 0.86
- Losing team average: 0.77
12% advantage to the teams that won.

- The team that won had the advantage 88 times (average of a 17% advantage)
- The team that lost had the advantage 28 times (average of a 7% advantage)



Death rate:
This is each pilot's cumulative deaths per match on the QP Leaderboard

- Winning team average: 65%
- Losing team average: 68%
4.2% advantage to the teams that won.



Matches played:
This is each pilot's cumulative matches on the QP Leaderboard

- Winning team total: 1,915,362
- Losing team average: 1,821,107
Teams that won play an average of 5.2% more QP matches



Tonnage:

- Winning team average: 65.0
- Losing team average: 65.2
0.3% advantage to the teams that lost.

- The team that won had the advantage 48 times (average of a 3.3% advantage)
- The team that lost had the advantage 68 times (average of a 3.2% advantage)



GMan Tier Rating
Lower is better. This is each mech's tier rating according to GMan129's site, MetaMechs. These tier lists are the opinion of only one player, so they are not absolute, but they are widely regarded as "pretty close."

- Winning team average: 2.1
- Losing team average: 2.2
5% advantage to the teams that won.

- The team that won had the advantage 63 times (average of a 17% advantage)
- The team that lost had the advantage 53 times (average of a 14% advantage)



GManTonnage
Higher is better. This is mech tonnage adjusted by GMan tier rating. A tier 3 mech will be worth its facevalue, while a tier 1 mech will be worth more tons and a tier 5 mech will be worth fewer tons. The formula I decided on is [ Tonnage + (12 * (3 - Tier)) ]

- Winning team average: 75.6
- Losing team average: 74.7
0.1% advantage to the teams that won.

- The team that won had the advantage 65 times (average of a 7.9% advantage)
- The team that lost had the advantage 51 times (average of a 7.1% advantage)



Clan vs. Inner Sphere
Number of each tech base per side.

- Winning team: 40% IS, 60% Clan.
- Losing team: 48% IS, 52% Clan
The winning team had 15% more clan mechs on average.



Extra tidbits:

The higher-match-score-team was the winner 88 times. Of the 28 times that it lost, the opposing team had the higher WLR 14 times. In fact, of the 28 times that the higher-match-score-team lost, 25 of them could be explained by the fact that the other team had either more tonnage, more meta mechs, or higher WLR. That leaves only 3 matches unexplained by variables that I investigated.

The higher-WLR-team was the winner 93 times. Of the 23 times that it lost, the opposing team had the higher-match-score-players 9 times. Also, of the 23 times it lost, 20 could be explained by the other team having more more tonnage, more meta mechs, or higher matchscore. That leaves only 3 matches unexplained by variables that I investigated.





CONCLUSIONS:

Players that tend to win more will more often be on the winning side of a stomp. (I KNOW, GROUNDBREAKING SCIENCE, RIGHT??!!) The team with higher collective WLR was on the winning side of a stomp 80% of the time. This was the strongest correlation of all the variables I examined.

The next strongest factors were QP average match score and kill rate. The team with higher match score was on the winning side of a stomp 76% of the time, and the team with the higher kill record also won the stomp 76% of the time.

Team tonnage had a negative correlation. The heavier team was on the losing side of a stomp 59% of the time. I suppose this makes sense if you look at it from the perspective that the team with the heaviest and slowest fatties is more likely to have those fatties get left behind in the Nascarfest and get picked off, initiating a snowball effect. I can't assert this fully, but if there is enough interest I could investigate a little more closely and see if I can confirm these suspicions.

Mech tier rating and mech worth-ness didn't seem to be an appreciably large factor, only in favour of the team that won 54% and 56% of matches respectively, but nevertheless in favour the slight majority of the time.




COMMENTARY:

These findings support my theory that either the PSR system or Matchmaker does not adequately assess the skill level of individual players. I fear that the Matchmaker only sees PSR Tier and nothing else. Here is an example:

Matchmaker sees available players:






Matchmaker splits them up into two teams:




Looks perfect! The weight classes are evenly filled, and there is an equal number of each Tier per team. But what the matchmaker wasn't considering, was the following:






Now that we've revealed each player's average match score, we see a massive imbalance! But if we just shuffle a few players from one side to the other:






Now it looks much more balanced! PGI, why can't your matchmaker do something like this?

Edited by Tarogato, 30 January 2017 - 01:47 PM.

[Monkey Lover]

Quote

Clan vs. Inner Sphere
Number of each tech base per side.

- Winning team: 40% IS, 60% Clan.
- Losing team: 48% IS, 52% Clan
The winning team had 15% more clan mechs on average.

Quote

GMan Tier Rating
Lower is better. This is each mech's tier rating according to GMan129's site, MetaMechs. These tier lists are the opinion of only one player, so they are not absolute, but they are widely regarded as "pretty close."

- Winning team average: 2.1
- Losing team average: 2.2
5% advantage to the teams that won.

- The team that won had the advantage 63 times (average of a 17% advantage)
- The team that lost had the advantage 53 times (average of a 14% advantage)

Looks like Gman tier ratings needs to move clan mechs up or IS mechs down a little

[RestosIII]

Huh. A surprising amount of work put into this, and was worth me taking the time to check out.


But on the other hand, this is going to be a good deal of a people when they see how much you wrote.

[Scurro]

Quote

Now it looks much more balanced! PGI, why can't your matchmaker do something like this?

It used to be like that with Elo. That was removed.

[Bud Crue]

Very cool.

I'm intrigued by the "Gman Tonnage" equations and aspect, particularly in that it seems obvious that tonnage is at best a secondary consideration both empirically and based on your data (negative correlation), yet PGI by word and deed seems convinced that tonnage more than most anything else is the most significant factor in determining balance and probability of wins (see group que tonnage restrictions, see attempts at addressing population imbalance by tonnage differences in CW, etc.).

Your study brings this to a head and I can't help but wonder why they insist on believing that absolute tonnage is what seemingly matters most?

[xTrident]

Very nice Tarogato, and I thank you for the time and effort you and your lance mates put in to this.

Your final analysis is great on the PSR and the matches. Basically the better players are always winning. That's the way I see it... All it would take is to split those "better" players up a bit to maybe even the matches out some.

All in all this is great info, thank you.

Edited by xTrident, 30 January 2017 - 02:37 PM.

[Tristan Winter]

Nice job.

I'm sure we've all had matches where some guy on your team was upset and said something like "That's why you should never push saddle" or "That's why you have to protect your assaults" or "That's why your team should never ever split up!" or some other infallible truth. Meanwhile, you're pretty sure that both teams did basically exactly the same thing, and that one side got roflstomped because the other side just had better players.

I know I have.

Edited by Tristan Winter, 30 January 2017 - 03:14 PM.

[Ultimax]

Nice work, although your conclusion kind of glossed over the big disparity your data shows when 1 team has more clan mechs.

Bud Crue, on 30 January 2017 - 01:58 PM, said:

Very cool.

I'm intrigued by the "Gman Tonnage" equations and aspect, particularly in that it seems obvious that tonnage is at best a secondary consideration both empirically and based on your data (negative correlation), yet PGI by word and deed seems convinced that tonnage more than most anything else is the most significant factor in determining balance and probability of wins (see group que tonnage restrictions, see attempts at addressing population imbalance by tonnage differences in CW, etc.).

Your study brings this to a head and I can't help but wonder why they insist on believing that absolute tonnage is what seemingly matters most?

This is really a tangled ball that would need to be unraveled.


Tonnage is huge, it is amazingly strong - and in the group QP queue its very clear many times when teams of similar skill face each other the one with a lot more tonnage usually has a huge if not insurmountable advantage.

It is one of the reasons game balance, mech balance & weapon balance here is a bit of "dartboard" - because PGI insists on using the lowest common denominators for their balancing.


Potatoes will potato. They think KDK-3 was an average mech, they think LRMs are good weapons (or they think they are overpowered), they think light mechs were/are overpowered.

None of that holds true once you have players who are actually good at the game square off, but that's what we have here very frequently.


Is tonnage powerful?

Yes.

Is it an automatic win?

Not if your Assault pilots are awful - and for some reason many awful pilots are drawn to assaults in the solo queue.

Edited by Ultimax, 30 January 2017 - 02:10 PM.

[Tarogato]

Tristan Winter, on 30 January 2017 - 02:04 PM, said:

Nice job.

I'm sure we've all had matches where some guy on your team was upset and said something like "That's why should never push saddle" or "That's why you have to protect your assaults" or "That's why your team should never ever split up!" or some other infallible truth. Meanwhile, you're pretty sure that both teams did basically exactly the same thing, and that one side got roflstomped because the other side just had better players.

I know I have.

Well that, and a couple brave smart player might go to a good location on the map, and the entire team follows.

But you have a couple dumb inexperienced players in front that everybody follows instead, and they go to a bad location on the map? Could be gg piecemeal.

[Khereg]

+1 for use of Durian.

+2 for placing him next to Onion, which he resembles so closely.

When we discussed this previously, what jumped out at me was the balance in pretty much all the measurable metrics. My personal tl;dr is that when you get stomped, your team played together poorly as a team while the other team did much better. If you've ever played comp, you know how much a couple of minor mistakes in teamplay can snowball into an embarrassing stomp.

Left your assaults? Went YOLO too early in that brawler? Got strung out while moving? Didn't share armor? ggclose, scrubs. Don't blame the matchmaker (except in jest, cause that's always funny).

[Little Details]

You take a while to get there, but your last point is really the one that matters - MM doesn't split players correctly between teams.

Most of us that have been playing a long time can look at the names on each side and probably have an 80% chance of guessing the match outcome before we even fall out of the dropship because we know who the good players are and when it's shifted to one side like your example, it's patently obvious what will happen in the next 6 minutes.

[Dimento Graven]

I think the tonnage correlation would indicate the team that's more mobile (what a theoretically lighter team would be) has an advantage over the more slowly lumbering team.

It would be interesting to figure out if the "heavier" teams were more weighted towards IS 'mechs too.

[Tristan Winter]

Tarogato, on 30 January 2017 - 02:15 PM, said:

Well that, and a couple brave smart player might go to a good location on the map, and the entire team follows.

But you have a couple dumb inexperienced players in front that everybody follows instead, and they go to a bad location on the map? Could be gg piecemeal.

Too bad it's hard to extract statistics about where the teams were moving, which positions are most likely to lead to victory on different maps, etc.

The equivalent of heatmaps / motion tracking for sports.

Spoiler

[Tarogato]

Dimento Graven, on 30 January 2017 - 02:18 PM, said:

I think the tonnage correlation would indicate the team that's more mobile (what a theoretically lighter team would be) has an advantage over the more slowly lumbering team.

It would be interesting to figure out if the "heavier" teams were more weighted towards IS 'mechs too.

I do want to break it up by weight class, and also break up IS vs Clan by weight class as well, and see if there's any patterns. But not today, I spent long enough typing this post up. =D

[Dimento Graven]

Tarogato, on 30 January 2017 - 02:20 PM, said:

I do want to break it up by weight class, and also break up IS vs Clan by weight class as well, and see if there's any patterns. But not today, I spent long enough typing this post up. =D

Don't blame you. I can wait for the results!

[Ex Atlas Overlord]

Bud Crue, on 30 January 2017 - 01:58 PM, said:

I'm intrigued by the "Gman Tonnage" equations and aspect, particularly in that it seems obvious that tonnage is at best a secondary consideration both empirically and based on your data (negative correlation), yet PGI by word and deed seems convinced that tonnage more than most anything else is the most significant factor in determining balance and probability of wins (see group que tonnage restrictions, see attempts at addressing population imbalance by tonnage differences in CW, etc.).

Anyone that's tried to fight a good light pilot while in an assault can tell that tonnage means jack **** currently in MWO.

Speed and smaller hitboxes are the real MVP.

Edited by Ex Atlas Overlord, 30 January 2017 - 02:23 PM.

[habu86]

Great post Taro. Always nice to see someone actually willing to invest the effort in crunching these numbers as opposed to just thinking about maybe, potentially gathering the data and getting this done.

I do have one issue with the post though.

Tarogato, on 30 January 2017 - 01:43 PM, said:

Now it looks much more balanced! PGI, why can't your matchmaker do something like this?

Bear with me on this, as it's been a long time and I don't recall the exact date, but I remember a town hall where Russ gave details about how the PSR matchmaker worked and, if I remember correctly, he specifically mentioned average numerical differences in tonnage and PSR (not Tier, but the actual PSR numbers) and how things had supposedly gotten much better after the last optimization pass. So, my understanding of it is that MM is already optimizing to minimize average PSR difference, with Tiers acting as constraint terms on how far it can reach to make things work (i.e. a Tier 1 and a Tier 5 player are not equivalent to 2 Tier 3 players).

And this, IMO, feels like a much bigger problem at times, where there seems to be a bigger discrepancy in performance between players of the losing team than the winning team.

Would you maybe take a pass at calculating KS distances or Gini coefficients in average match scores between losing teams and winning teams in stomps vs. more closely fought matches?

Edited by habu86, 30 January 2017 - 02:28 PM.

[Milocinia]

A thoroughly amazing and comprehensive look at matchmaking. I dread to think how long it took you to crunch all of that data.

While it's only a small sample size, there does seem to some clear results showing through and it would be nice if PGI could act on it and use it to improve the matchmaker.

But they won't.

[Khereg]

LT Satisfactory, on 30 January 2017 - 02:16 PM, said:

Most of us that have been playing a long time can look at the names on each side and probably have an 80% chance of guessing the match outcome before we even fall out of the dropship because we know who the good players are and when it's shifted to one side like your example, it's patently obvious what will happen in the next 6 minutes.

Agreed, but I view it as a corollary to what I said - we know the "good" players in the sense that those players both have individual ability and also aren't likely to do something boneheaded from a teamplay perspective.

I've been surprised, though. I recall a particular match a couple of weeks ago where virtually every player on both teams had name recognition for being unspud-like. I thought, 'hey. this should be interesting".

Domination on Crimson. My team literally failed to move to the circle until I raced in with less than 5 sec on the clock with a PPC HBK (which is decidedly NOT what you do with a PPC HBK). To no one's surprise, I got killed in 30 seconds by a couple of light brawlers while our other team members trickled in and got killed one-by-one. It was the most talented Tier 5 match I had ever seen.

I chalked it up to the idea that everyone was used to hanging on the outside during domination while the lower tier players held the circle and it backfired when literally EVERYONE tried to do it. Scrubs. All of us.

[Tarogato]

Ex Atlas Overlord, on 30 January 2017 - 02:22 PM, said:

Anyone that's tried to fight a good light pilot while in an assault can tell that tonnage means jack **** currently in MWO.

Speed and smaller hitboxes are the real MVP.

It's not quite that though. We're talking about cumulative tonnage in solo queue, which is a roughly even distribution of all four weight classes. So factors are things like... Awesome vs Kodiak, and Cicada vs Stormcrow.

While a Locust can easily solo kill many assault mechs, you can't deny that having 800 tons is stronger than having 700 tons.

Also, it's my hunch that a lot of the higher-tonnage team's losses can be attributed to other factors (players' predispositions toward winning and scoring higher) overshadowing the effect of tonnage. I'll have to delve deeper.