8
For most recent analysis.
Someday, I'll restructure this first post to contain everything.
Original Post Below:
A statistical analysis of Mechwarrior Online leader board data. All data are from mwomercs.com.
You have been warned. This will be a wall of text with some charts and graphs...
I did this because I was both bored and frustrated with the continual sniping about which is better (IS or Clan) and balance in the game. Lots of anecdotal evidence on both sides, but surprisingly little real analysis. The actual analysis didn't take very long. Extracting the data and doing the minimal post run graphics (note the low effort on the pictures....)
Synopsis:
In short, overall clan mechs have a distinct and statistically significant advantage over IS mechs based on scores. Both IS and Clans have some mechs that hold advantages at the top end of the range. IS is overwhelmingly represented at the bottom end of the performance range and Clan mechs have more mechs in the top end of the ranges than the IS, some of them by large margins.
A small portion of this work was informed by a spreadsheet prepared (I'm told) by Tarogato from ISEN (Thank you). I recompiled the data on my own, though I reused his approach for integrating data for the Viper.
I'll probably update this to include the Cyclops after it has a leader board event and after the heavy event assuming that there haven't been too many changes in the game or scoring.
All data and processing are available at: https://github.com/n...MWOLeaderBoards
All results and any opinions are my work only. Please provide credit should you choose to build on my work.
Some notes and thoughts before I dive in.
The distributions of scores are non-normal and do not transform easily, so all of these results should be interpreted with caution. i.e. I believe the conclusions we can draw are valid, but if you're going to base an active handicapping system or online gambling odds on these you're on your own (unless the latter proves profitable, in which case I want royalties).
I'm making a number of other assumptions here that should be stated.
1. The top 75 results for each mech are representative of high performing pilots in that mech and that the pilot quality is even between mechs. i.e. we don't have a major effect in which mech rankings are influenced by better pilots choosing to pilot better mechs leaving poorer pilots to dominate the lower performing mechs.
2. That the scoring system is a good representation of the utility of the mech.
3. The non-normality of the data isn't a critical failure (I don't think it invalidates the results, just raises some questions where results are close).
4. A linear model is appropriate (It generally tests well and has reasonable distributions of residuals for the model), but given the non-normality is subject to being questioned.
Point 1 is probably the biggest reason to doubt the underlying message behind the results. The others either are uniform across mechs, or may have marginal effects that slightly change orders of mechs that are close to each other. We could debate this for pages, but I'm going to (non-scientifically) rationalize that if high performing pilots are disdaining the use of certain mechs, it's because they feel that those mechs put them at a competitive disadvantage (reinforcing the point, though not in a quantifiable way).
High scores can result from two primary mechanisms. High raw damage output, or high longevity on the field. There's no good way to quantify the degree to which each influences the score from the data used.
Reporting on Statistics:
Every statistic generated is significant with p<0.001, mostly because of the number of samples (>3000 as of this writing).
The dataset used includes the results of the Light, Medium, and Assault leader boards, and the to 75 results from the viper leaderboard. (see the github account for the xls and csv used to prepare the data).
IS vs Clan Expectations:
In a linear regression model that accounts for the mech's tonnage and whether it is IS or Clan tech the resulting prediction for score is: (IsIS = 1 for IS mech, 0 for Clan)
Score = Tons*9.84 - 239.1*(IsIS) + 2139.3
R-Square: 0.269
Interpretation: Across the entire range being in a clan mech adds about 240 points to your score. Given the range of scores, that ranges between about 6.5 and 12.5% of the final scores.
Using this function and computing residuals for each result, we can then look at which mechs perform above and below the expectation. Ranked by mean residual:
The locust is the highest performing IS mech relative to tonnage, but still lags behind several clan mechs. The Mist Lynx is the lowest.
Mechs by Tonnage:
The above example assumes that clan mechs will outperform IS mechs on average. Given PGI's statements on balancing, and the lack of other limitations on clan mechs (either in numbers deployed to the field, penalties for dishonorable combat, and employing mercenaries), I don't believe that to be the intent.
If we compare purely on the mech's tonnage we get the following regression model.
Score = 9.98* Tons + 1975.4
R-Square: 0.221
The plot of this function against the input data:
Note high performance at some weight classes (20 and 40 tons most notably).
Using the same analysis of the residuals:
If we assume that there should generally be ton for ton parity between IS and Clan mechs, at least as measured by the scoring system used in the leader board, we have a clear failure. Of the top 10 performing mechs by tonnage, 3 are IS, and the top 4 are all clan mechs and by large margins. Similarly at the bottom of the list, of the bottom 10, 1 is Clan. And the worst performers are at the bottom by a pretty substantial margin.
PS. Edited for formatting. I'm sure there are other typos and updates.
If PGI would like me to do more work by giving me greater access to their data, I'm willing and able. I have a significant background in the statistical modeling of behavioral and geographic phenomena.
Also added R-Square's for the regressions
Edited by Tahawus, 28 September 2016 - 07:48 PM.
