85 replies to this topic

[WANTED]

More love for this. Hard data is good to expose

[Dawnstealer]

How dare you suggest the Urbanmech isn't overpowered. HOW DARE YOU.

[Tahawus]

Dawnstealer, on 23 August 2016 - 10:58 AM, said:

How dare you suggest the Urbanmech isn't overpowered. HOW DARE YOU.

Don't shoot the messenger.

No, really, don't, I'll probably be in my beloved Phoenix Hawk, and I'd really like to keep that arm.

[habu86]

Boogie138, on 21 August 2016 - 11:48 AM, said:

How did you account for pilot skill?
-giggles-

Tahawus, on 21 August 2016 - 12:39 PM, said:

I didn't except as it is represented in the error term for score. See my discussion on point 1 for related info.

Go to the in-game leaderboards to get some of that data. We know who the top 75 pilots on each weight leaderboard are and you can look up their recent stats by weight class in-game. No need to worry about one-game wonders here either. They've all had to put in legitimate work to make the leaderboards.

It'll be a bunch of work to get everyone's stats across 5 categories, but I think it might be worth doing so for a number of reasons:
1.) it would help us get an idea of whether skill is, in fact, randomly distributed - my guess is that, at that level, there are a few true standouts in terms of skill (aka freaks of nature, and I mean that in a good way), with most everyone else being relatively more closely matched
2.) to the extent that those metrics are reflective of actual skill, they can be introduced into the modelling dataset to control for some of it
3.) it would help us identify the standouts and control for them. There are few names on that list that, IMO, legitimately warrant their own dummy variables

P.S. love the initiative. wishing to hell that PGI would let us download match-end data to better analyze this stuff.

Edited by habu86, 23 August 2016 - 12:27 PM.

[Tahawus]

Habu,

I'd love to, but manually search, and transcribing pilot into from the game client for >3000 records with (I'm guessing) more than 500 unique pilots would cut into my mech shooting time.

If they were to give is an API...

[Flashover23]

Tahawus, on 23 August 2016 - 08:08 AM, said:

They might also consider a committee of data analysts drawn from the community as a review panel for balance.

They can consider all they want, while continuing to do the same thing they have before with varying results.
(Yes, I'm trying to be polite. It's a new thing for me, too.) But all the consideration and pondering and brainstorming won't do anything without implementation of the idea(s).

They need to look at data/information a new way, and not become enamored with the "not-invented-here" mindset.

Otherwise, they're just sitting at their desks "considering" a larger right bicep.

[habu86]

Tahawus, on 23 August 2016 - 12:40 PM, said:

Habu,

I'd love to, but manually search, and transcribing pilot into from the game client for >3000 records with (I'm guessing) more than 500 unique pilots would cut into my mech shooting time.

If they were to give is an API...

I hear you. I'd do it myself, in the interest of science, but am doing something very similar for work and it's burning me out on this sort of stuff.

Manual data cleaning and conforming? In this day and age? Oh the indignity...

[Asaru]

This kind of data is great and if we could get more like it from PGI I feel it would be a huge benefit for the next FW Round Table.

[Tahawus]

Just because I like playing with data...

A lot of this has already been beaten to death, but I'm a visual kind of person...

Important note: because of the nature of this leader board, we only know the dominant mech used by each pilot. This raises a lot of questions and should leave us using the data with caution. That said, the results are a strong indication of clan mechs being dominant on this leader board.

Basically, the Clan and IS are relatively balanced in terms of total representation in the top 75 of the heavy mechs. But, the median position of Clans is 31, and of IS 46. This indicates that the clans have a positional advantage in heavies.

All of the other weight classes aren't even close. Clan mechs dominate them.


Lights:

Clan:	46
IS:	26

Adder		Clan	2
Arctic Cheetah	Clan	38
Jenner		IS	9
Jenner IIC	Clan	5
Kit Fox		Clan	1
Locust		IS	13
Raven		IS	4
Spider		IS	1
Urbanmech	IS	1
Wolfhound	IS	1

Mediums:

Clan:	55
IS:	20

Blackjack	IS	2
Centurion	IS	4
Cicada		IS	4
Crab		IS	1
Griffin		IS	6
Hunchback	IS	1
Hunchback IIC	Clan	19
Nova		Clan	13
Shadow Cat	Clan	6
Shadowhawk	IS	2
Stormcrow	Clan	16
Viper		Clan	1

Heavies:

Clan:	39
IS:	36

Black Knight	IS	2
Cataphract	IS	1
Catapult	IS	5
Ebon Jaguar	Clan	13
Grasshopper	IS	1
Hellbringer	Clan	6
Mad Dog		Clan	8
Marauder	IS	6
Orion IIC	Clan	1
Quickdraw	IS	1
Rifleman	IS	1
Summoner	Clan	3
Thunderbolt	IS	1
Timber Wolf	Clan	8
Warhammer	IS	18

Assault:

Clan:	64
IS:	11

Atlas		IS	3
Awesome		IS	1
Battle Master	IS	2
Dire Wolf	Clan	14
Executioner	Clan	2
Highlander IIC	Clan	3
King Crab	IS	3
Kodiak		Clan	42
Mauler		IS	1
Stalker		IS	1
Warhawk		Clan	3

Edited by Tahawus, 23 August 2016 - 09:42 PM.

[Tahawus]

Morning bump for the fast moving forum.

[exiledangel]

tahawus love the spread sheets really easy to fallow. Now about the Kodiak ( in my personnel opinion) if the Kodiak was available for C-bills the leader board would have been just Kodiak true and true because a lot of the MechWarrior don't want to spend Real money on mechs they can wait for 3-4 months down the road. now with that taken in to account if you messaged each player that finished the leader board with IS mech and asked them if they possessed a Kodiak the answer would probably be No didn't want to spend the money. Remember I used the word probably which means some ppl bought and went meh I go with what I feel comfortable piloting.

[Tahawus]

Bump to keep this on the radar, and in the hope that it can contribute to the discussion about balance from an information based perspective.

[4EVR]

So what I take away from this is that clans aren't OP, the Kodiak is perfectly balanced, and the Ebon Jag isn't better than the Timby.

/ggclose


PS. How about power rankings with confidence intervals? Kodiak mean = 100.

[Tahawus]

A bit more. Now including the Cyclops.

But first some cautions about interpreting the results for the Cyclops and the Viper. I grow increasingly less convinced that they should be included here. Within these stats, both seem to outperform the sentiment of those I've talked to. I think there are several reasons for this. First, and foremost, it's the only chassis in the leader board and all players are focusing on playing it. This means that among other things some elite players who can make any mech look good are represented multiple times in the top 75 scores for the one chassis. The Viper and Cyclops leader boards have been slightly longer than the others allowing a little more upwards creep.

The chart of residuals as done previously.



And based on reader suggestion, using a Log-Log transformation to address some of the outlier issues. Note that this does change the ordering slightly of some of the mechs with more outliers, notably at the very top off the ranking.

I belive I'm interpreting this correctly, but would appreciate some of the folks with prior Log-log interpretation's feedback. Essentially, the residual in the Log-log represents a percentage advantage or disadvantage (i.e. The Arctic Cheetah's 0.248 indicates that it over performs what it should for it's tonage by 24.8%).

Overall, while the specifics have changed slightly, the take home message hasn't changed much. The high performers remain the high performers with minor reordering, and the bottom of the heap remains at the bottom. Clan mechs dominate the top of the performance spectrum, and IS mechs make up the majority of the bottom end.

[Joey Tankblaster]

More love for this ...

~bummp~

[Bud Crue]

This is amazing work. Well done Tahawus. A service to the community.

[Kyp Durron]

FupDup, on 21 August 2016 - 11:49 AM, said:

I don't like the "score per ton" statistic because it implies that choosing a smaller mech should instantly make you inferior to somebody who chose a bigger mech.

So you rebut data with an emotional rejection?

It's possible that this data shows that there's an imbalance in the game which is being reflected by mech size. It's possible the game isn't scaled properly for lighter mechs to adequately perform.

[Dogstar]

I'm very impressed with this as it nicely verifies my personal opinions. It's very curious how some obviously badly performing (mostly IS) mechs aren't improved and how the top end (Clan) mechs don't get hit by the nerf bat.

The game is lacking in balance, more so than could be explained purely by the imbalanced table top inheritance.

[Andi Nagasia]

Kyp Durron, on 07 September 2016 - 06:59 AM, said:

So you rebut data with an emotional rejection?  

It's possible that this data shows that there's an imbalance in the game which is being reflected by mech size.  It's possible the game isn't scaled properly for lighter mechs to adequately perform.

i dont think this data tells the full story, not that its not accurate but it doesnt have all the variables,
for instance i expect the AWS / DWF / KDK to score very high, on an all assault Event,
as their are lots of Large Mechs with lots of health, but dont expect that same DWF to score high in a Light Event,
also the sample size as stated by OP is a huge factor, if more people are playing KDKs KDKs should have a better top 75,

Edited by Andi Nagasia, 07 September 2016 - 07:10 AM.

[Grinster]

Tahawus, on 04 September 2016 - 07:52 PM, said:

A bit more. Now including the Cyclops.

But first some cautions about interpreting the results for the Cyclops and the Viper. I grow increasingly less convinced that they should be included here. Within these stats, both seem to outperform the sentiment of those I've talked to. I think there are several reasons for this. First, and foremost, it's the only chassis in the leader board and all players are focusing on playing it. This means that among other things some elite players who can make any mech look good are represented multiple times in the top 75 scores for the one chassis. The Viper and Cyclops leader boards have been slightly longer than the others allowing a little more upwards creep.

[snip]

And based on reader suggestion, using a Log-Log transformation to address some of the outlier issues. Note that this does change the ordering slightly of some of the mechs with more outliers, notably at the very top off the ranking.

I believe I'm interpreting this correctly, but would appreciate some of the folks with prior Log-log interpretation's feedback. Essentially, the residual in the Log-log represents a percentage advantage or disadvantage (i.e. The Arctic Cheetah's 0.248 indicates that it over performs what it should for it's tonage by 24.8%).

Overall, while the specifics have changed slightly, the take home message hasn't changed much. The high performers remain the high performers with minor reordering, and the bottom of the heap remains at the bottom. Clan mechs dominate the top of the performance spectrum, and IS mechs make up the majority of the bottom end.

[snip]

Thank you so very much for all your work. There is a lot of data produced in every match and a proper analysis should inform decision-making by PGI and confirm or deny player's claims of chassis performance differences.

The stats available in the profile page for players is also a potential source should we want to pool them. Your Cyclops results qualitatively match with mine as far as it being the strongest mech in my IS stable and it is pleasing to see the Awesome up there.

There are a lot of gems to be gleaned from the personal stats such as: Conquest is the best for C-Bills; we all need to play Terra Therma more!