57 replies to this topic

[Tarogato]

Data: https://docs.google....dit?usp=sharing

^ If you feel you can do something more with it, go for it!

Want to help by submitting your own stats? --> I'll be posting another thread in the coming months to collect more data and compare.

---

Tally as of 9/29/15:

201 players collected = sample size.
The average player is Tier 2 (~63% bar filled).
The average player has played 3,903 matches.
The average player has a 1.59 KDR and 1.29 WLR.

Tier 1: 48 players submitted (24%)
Tier 2: 59 players submitted (29%)
Tier 3: 61 players submitted (30%)
Tier 4: 19 players submitted (9%)
Tier 5: 13 players submitted (6%)

25 players reported being maxed out Tier 1 (12%, exactly half of all Tier 1's)

Of course, the above data may not accurately reflect the actual whole MWO population, as I collected volunteers and higher tier players are more likely to be on the forums/reddit to see my post and volunteer. However, it is enough data to plot out some charts and eyeball the trends within each tier - perhaps giving us a little insight on how well PSR is working as of this date compared to how well we would expect it to work.

---

R² = 0.07






R² = 0.05






R² = 0.47






R² = 0.36






R² = 0.18

Edited by Tarogato, 30 December 2015 - 04:42 PM.

[Kiiyor]

This is truly, truly excellent. Some of those outliers on the KDR/WLR chart are very interesting indeed.

If I could suggest anything, it would be to maybe start gathering information like 'dmg/match' etc, or even dmg/match and KDR for the top two or three mechs in a player's stable. That might shed some insights into the lower tiers - because a few that I know have respectable stats in a few stalwart mechs, and some pretty terrible stats in others.

KDR does eventually even out to be an indication of skill, but it depends a lot on the player also. Personally, I prefer to move to fresher opponents if I disable or drastically reduce a foe's ability to fight, and this reflects upon my KDR. While the same argument can be made about damage (what if it's spread out all over the place?) I think it's a far more reliable stat to use to track performance with - especially when it makes up such a significant portion of a player's match score.

What I would love to see, is counts of 'most damage done', 'solo kills', 'flanking' etc. added to stats. I've already petitioned the devs about it, but I might have to start pestering them on twitter

Again, great work.

[Hit the Deck]

Thanks for your work! I'd like to ask that you display the R² value on your graphics.

[Trevelyas]

Oh, look at that nice strong bias toward Tier 1 and the saturation at the max. This is what happens when you don't use a zero-sum calculation in your rating system.

Good luck to the match-maker trying to assemble competitive games using values from this "clamped integral" instead of a proper distribution.

Edited by Trevelyas, 29 September 2015 - 09:52 PM.

[meteorol]

Interesting. My matches played would seed me t3, my w/l is just above t2, and my k/d is considerably higher than t1 average.
Guess i don't really fit in.

[Hit the Deck]

Trevelyas, on 29 September 2015 - 09:48 PM, said:

Oh, look at that nice strong bias toward Tier 1 and the saturation at the max. This is what happens when you don't use a zero-sum calculation in your rating system.

Good luck to the match-maker trying to assemble competitive games using values from this "clamped integral" instead of a proper distribution.

It could also because instead of making a "blind poll" at first, he asked people to post their stats.

[lpmagic]

great work indeed, thanks for putting the time in to this.......some very interesting results, leads me to believe that specific metrics are attached to match score and not real world stats in general, very interesting indeed thanks again man, this is way cool!

[Sarlic]

Very interesting to see.

Thanks for your time, Tarogato!

Edited by Sarlic, 29 September 2015 - 10:02 PM.

[Duke Nedo]

Could you please plot (nr of wins) - (nr of losses) vs psr? Think there may be a very good correlation there...

Edited by Duke Nedo, 29 September 2015 - 10:05 PM.

[STEF_]

Interesting.

I think total players are about 20k (from last tournament numbers), so the sample is very small, and also forumers are the ones really committed with the game (and have supposedly higher tiers), while the vast majority (steering wheels) don't even entered the forum once.

I think I can estimate tier 4 and 5 population to be at least 10k.

It would be interesting to see real numbers, aka pgi showing % for each tier population.

[jay35]

I find it interesting that the first three charts appear to show a pretty nice distribution of stats across the tiers, implying the tiers are actually pretty good, despite us knowing that's not the case for anyone who spends most of their time playing CW or anyone who simply hasn't played the game much in the past six months.

Kiiyor, on 29 September 2015 - 09:36 PM, said:

This is truly, truly excellent. Some of those outliers on the KDR/WLR chart are very interesting indeed.

The odd-looking distribution on the KDR WLR corollary chart is primarily the result of 2 years (since the stat reset) (and, January through August, so still the vast majority of the data affecting PSR as well) of a matchmaker that tried to force everyone toward a 1:1 WLR. This is why the previous chart showing an almost complete lack of correlation between matches played and KDR looks so different. When we take WLR out of the equation, we see a much more natural widespread distribution.

And some of us suffered under that stupid WLR-focused matchmaker in solo queue for three years of grinding out mech chassis, and only recently emerged at the top of the stack (260+ individual mech variants ground out) and started playing mechs we actually enjoy and more group queue than solo queue. But we're saddled with years of data created while grinding, so our natural KDR is actually suppressed. It's nice that it's finally rising but it takes forever because there is such a mountain of matches already played so each new match played moves the KDR needle very, very little.

Edited by jay35, 29 September 2015 - 10:21 PM.

[Tarogato]

Hit the Deck, on 29 September 2015 - 09:44 PM, said:

Thanks for your work! I'd like to ask that you display the R² value on your graphics.

Ah frack, that's me being a total statistics noob. I've added it below each image now, thanks! If you think they'd be more valuable with outliers removed, let me know.

Duke Nedo, on 29 September 2015 - 10:02 PM, said:

Could you please plot (nr of wins) - (nr of losses) vs psr? Think there may be a very good correlation there...

Although many did, not everybody gave me their exact number of wins and losses each. I was indeed planning on going back and grabbing that information later, though. Will probably be a few days before I get around to it, though - if I do at all.

jay35, on 29 September 2015 - 10:19 PM, said:

primarily the result of 2 years ... of a matchmaker that tried to force everyone toward a 1:1 WLR.

This, I believe is a huge misconception about how the previous matchmaker worked. It did *not* try to force everybody to a 1:1 WLR, it simply tried to build matches where the average Elo of Team 1 was as close to Team 2 as possible.

Edited by Tarogato, 29 September 2015 - 10:31 PM.

[Sarlic]

Dat correlation..!

Edited by Sarlic, 29 September 2015 - 10:27 PM.

[Siriothrax]

Kiiyor, on 29 September 2015 - 09:36 PM, said:

This is truly, truly excellent. Some of those outliers on the KDR/WLR chart are very interesting indeed.

Yeahhh, that one in Tier 3 is me. He excluded me from the W/L graphs. Don't really feel like grinding to Tier 1.

[Varhait]

Thank you for this research. Very interesting results. But i can't understand one thing, what R2 = 0.07, R2 = 0.05, R2 = 0.47, R2 = 0.36, R2 = 0.18 means ?

[Trevelyas]

Siriothrax, on 29 September 2015 - 10:47 PM, said:

Yeahhh, that one in Tier 3 is me. He excluded me from the W/L graphs. Don't really feel like grinding to Tier 1.

This is precisely why I didn't report my own stats. I didn't want to muddy the data with a huge outlier.

[The Great Unwashed]

Varhait, on 29 September 2015 - 10:50 PM, said:

Thank you for this research. Very interesting results. But i can't understand one thing, what R2 = 0.07, R2 = 0.05, R2 = 0.47, R2 = 0.36, R2 = 0.18 means ?

http://www.statistic...on-coefficient/



It's a basic estimator that indicates how well one variable is linked to another. There's a lot of uncertainty in the MWO analysis as the player base is now quite small, though several charts show a nice cumulative probability function and with a lot of overlap per tier.

Edited by The Great Unwashed, 29 September 2015 - 10:59 PM.

[Siriothrax]

Trevelyas, on 29 September 2015 - 10:52 PM, said:

This is precisely why I didn't report my own stats. I didn't want to muddy the data with a huge outlier.

Yeah, well, I'm an ass. ^.^

[Tarogato]

Trevelyas, on 29 September 2015 - 10:52 PM, said:

Siriothrax, on 29 September 2015 - 10:47 PM, said:

Yeahhh, that one in Tier 3 is me. He excluded me from the W/L graphs. Don't really feel like grinding to Tier 1.

This is precisely why I didn't report my own stats. I didn't want to muddy the data with a huge outlier.

While outliers gonna lie out, they do at least reveal one of the weaknesses of the system. It takes many many games (I think Paul said it could be as much as a thousand) for a player to level out in the tier they deserve. In my opinion, a player should be properly rated well within his first 100 games.

Siri, have you probably played 100 games since PSR in January? Maybe more than 50? Or literally zero?

[Kmieciu]

I've heard US Scientists discovered that in order to loose weight, you have to eat less...

The biggest factor in PSR is winning. It was openly stated by Paul.

No wonder W/L ratio is correlated. And killing enemies is closely correlated to winning, because in 95% of the matches you kill the opposing team in order to win.

Edited by Kmieciu, 29 September 2015 - 11:45 PM.