85 replies to this topic

[Tahawus]

I am glad to see this continues to be of interest.

[Requiemking]

Tahawus, I understand your insight. However, I must ask you a question. What is it you feel makes these mechs OP, and did you account for specific variants/Omnipods, such as with the Kodiak?

Edited by Requiemking, 10 September 2016 - 06:05 PM.

[Mechwarrior Buddah]

tldr? rofl j/k

[Khobai]

clearly the locust needs a nerf

[Tahawus]

Requiemking, on 10 September 2016 - 06:05 PM, said:

Tahawus, I understand your insight. However, I must ask you a question. What is it you feel makes these mechs OP, and did you account for specific variants/Omnipods, such as with the Kodiak?

I consider a mech OP if it performs noticeably better than its predicted performance based on it's tonnage. i.e. it does substantially better than it should for a mech of XX tons.

Based on the most recent analysis, I feel pretty comfortable saying that the Arctic Cheetah ,Nova, Kodiak, Locust, Hunchback IIC, Cicada, and Dire Wolf are noticeably over powered as defined by having 10% greater performance than a mech of their weight should. The Cyclops and Viper have statistics that would put them into the OP category but the data was collected differently and I have my doubts about whether those data should be included.

As I've noted previously in this thread, there is no detail on variants (or omnipods) in the weight class leader board results and as such there are no details in the results presented here. I have not, but could do a similar analysis focused on the Cyclops and Viper chassis individually.

[Requiemking]

Tahawus, on 10 September 2016 - 07:42 PM, said:

I consider a mech OP if it performs noticeably better than its predicted performance based on it's tonnage. i.e. it does substantially better than it should for a mech of XX tons.

Based on the most recent analysis, I feel pretty comfortable saying that the Arctic Cheetah ,Nova, Kodiak, Locust, Hunchback IIC, Cicada, and Dire Wolf are noticeably over powered as defined by having 10% greater performance than a mech of their weight should. The Cyclops and Viper have statistics that would put them into the OP category but the data was collected differently and I have my doubts about whether those data should be included.

As I've noted previously in this thread, there is no detail on variants (or omnipods) in the weight class leader board results and as such there are no details in the results presented here. I have not, but could do a similar analysis focused on the Cyclops and Viper chassis individually.

Ok, but nerfing an entire chassis is not necessarily the best option. For example, it is a well known fact that the Kodiak 3 is OP. However, this is not reflected on the leaderboards. As such, your data, while valuable, is incomplete, and balancing around incomplete data creates a headache of problems when mechs that are balanced get hit with an unnecessary nerf. Case in point, the Jenner. The rescale nerfed the Oxide, just like everyone wanted, but the other variants, which were already balanced and didn't need to be nerfed, got nerfed anyways. Whats more, they got nothing in compensation, which is why the most of the lights in-game are Arctic Cheetahs and Locusts.

[Andi Nagasia]

Tahawus, on 10 September 2016 - 07:42 PM, said:

Based on the most recent analysis, I feel pretty comfortable saying that the Arctic Cheetah ,Nova, Kodiak, Locust, Hunchback IIC, Cicada, and Dire Wolf are noticeably over powered as defined by having 10% greater performance than a mech of their weight should. The Cyclops and Viper have statistics that would put them into the OP category but the data was collected differently and I have my doubts about whether those data should be included.

i still think the LCT, CDA, NVA, DWF, results are Skewed,
the LCT & CDA can do amazing damage if they are ignored, which they aften are,
and the NVA & DWF are Gun Boats that again can push Huge Numbers if they arnt Focused First,

each has its Problems,
is much easier to have a Bad game(sub100Damage) with those 4 Mechs than a Great Game(+1000Damage),
especially in Pug Que, where you dont have back up & cant depend on much more than your self sometimes,
-
in group Que people usually take whats effective, and thats usually not the LCT or DWF,
the LCT cant command enought damage to be Valuable to the team, and the DWF way too slow & easy to hit,

the problem with these Events, and the current rewards system is Damage is most Important,
and that can usually be inflated in some in stances, for instance when you Drop in a Match with lots of Assaults,
if your Mech is Geared to Dakka Focusing and Luck has them come at you one at a time 1000damage easy,
-
and most Events as People Know big rewards come with Big Damage,
and Players also know Big Assaults Bring in the Big Damage Numbers,
so its usually assaults on assaults inflating Damage Numbers even more,

not that the data you Collected isnt Accurate, but i think there are more Variables to Consider,
perhaps looking at the #200 Ranked to #300 Ranked People would show a more balanced Values?

[Tahawus]

At no point have I suggested that my work should be the basis of a rebalancing. Doing so would, as you suggest, be a misuse of the results. They are not detailed enough to identify what aspects of each variant should be modified to create a more balanced system.

I also recognize and agree with the sentiment that the scoring function may not be the best measure of overall utility. I'd like to see it balanced with respect to kills, kmdd, and component destruction than damage.

What we've got is a replicable method for analyzing the mech's performance that can be used as a quantitative measure of relative performance. The methods I've demonstrated can be used with any scoring system so long as it is consistent across the entire data set being examined. i.e. take the same scoring rubric (or develop a new one), but apply it not to the top 10 matches, but to the last 10 (or 20, or...) then apply these methods.

This analysis could be extended easily from chassis to variant, and with some more work omnipods if the data is made available. It'd be great to get more pilot information in as an additional control.

An analysis of a larger data set with appropriate scoring, could be used to identify which variants over and under perform, and then be used again to evaluate the success of balancing efforts.

As it is, what I can unequivocally say, is that based on the data made available through the leader boards, those chassis listed above provide at least a 10% advantage in scoring over a hypothetical "average" mech of the same weight. In this game, 10% is a meaningful margin, particularly when most of the most benefitted mechs are clan, and more of those are heavier than those that over perform from the IS. I.e. the 23.4% bonus for the Kodiak probably (note: I don't have a way to test this) makes a bigger difference in matches than the 23.5% of the Nova and certainly more than the 21% of the Locust.

[Captain Mittens]

The problem with this data set is that these scores only reflect the very top matches played by the very top players.

This reflects only a measure of highest possible score in a match. This will always favor high risk/reward builds like snipers and lrm boats because they might be shooting all match and not get taken out because of the meat in front of them. Brawlers rarely make it to the super high score because they take a lot of damage. Locust is high because you can have a match where you're just in the zone and the enemy is sleeping and you shoot assaults in the back all Day.

These scouts don't reflect the number of times a sniper gets caught by lights and leaves the match with 30dmg dealt.

[Razorfish]

Love your post Tahawus. I find your conclusion well thought out and correct.

All the others on this thread who don’t think Clan mechs, as a whole, far out perform IS mechs… Please put the crack pipe down and step away.

[Tahawus]

And now for the inevitable update to include the heavy leader board.

NOTES AND CAVEATS
1. No, I'm not recommending re-balancing solely based on this.
2. Yes, I'm aware that leader boards are not an ideal performance measure, far from it.
3. There is no better publicly available data set that I'm aware of. If you are aware of one, please point it out.
4. If PGI want's to share more disaggregate data, I'll take vacation from work to do analysis on it.
5. I've dropped the Cyclops and Viper out (and never included the Night Gyr) of this version of the analysis. I don't think their variant level format is a good comparison with the other mechs even when sorted to the top 75 performers across all variants.
6. Yes, there are a lot of reasons these data are not ideal. Sampling issues, pilot bias, slightly varying collection windows, participation, debatable scoring metric, and others.
7. Statistics lie, but I've done my best to make it so that you can police my work.
https://github.com/n...MWOLeaderBoards

But, when you get down to it, this is the best data collected under similar circumstances for a large variety of mechs, and if we're going to discuss (note I said discuss not argue) which mechs may be over or under powered compared to the entire population, let's do it with some quantifiable metrics as a component of the discussion.



RESULTS
Overall, results close to expected, but there are some important notes. The results of the heavy leader board were substantially messier (higher variability and less predictability based on tonnage) than the others and have resulted in some changes to predicted values and orders from the other classes, In particular, when the heavy mechs are included, it pulls down the predictions for assaults resulting in several assault mechs having relatively higher performance compared to the prediction.i.e. many assault mech leapfrog up the standings past some of the mediums and lights as reported previously.

The following table is based on the model formulation:
ln(score) ~ ln(tons) + error

The results are highly statistically significant (p<0.001), but that's to be expected from a 5025 record data set being tested on one variable.
R-Square in this case is ~0.16, meaning that the mech's tonnage explains about 16% of the variation in the score.

edit: the second column (ln(score)) should actually be mean(ln(score)), but I'm not going to redo the screenshot.



For what it's worth ln(score) ~ ln(tons) + (ISvsClan) + error is also highly statistically significant for both tons and ISvsClan, in the expected direction (i.e. an IS mech is on average expected to produce a lower score than a clan mech of equal tonnage). R-Square for this model is ~0.20. So the addition of ISvsClan improves our explanatory power by about 4 percentage points.

DISCUSSION
While orders have changed slightly, the take home message remains generally consistent. Based on the scoring rubric and structure of the leader boards clan mech outperform IS mechs by a detectable and statistically significant margin. With the inclusion of the heavy mechs the clear and absolute superiority of clan mechs erodes somewhat.

Of the top ten performing mechs: 6 are clan, 4 are IS.
Of the top five: 4 are clan
The top 3 are all Clan, and have a substantial lead over the next mech (Locust) in performance per ton.

Of the bottom ten mechs: 9 are IS
The bottom five are all IS
The lowest Clan mech (Orion IIC) is 7th from the bottom.
Positions 11 and 12 from the bottom are the Myst Lynx and Summoner respectively.

A sizable majority of the mechs are reasonably balanced. For example 42 of the 67 chassis evaluated have residuals between 0.1 and -0.1. On each end of the spectrum there are 5-6 mechs that I'd assert are extreme values and should be considered for adjustment by PGI.

Edited by Tahawus, 28 September 2016 - 08:03 PM.

[Big Tin Man]

Well done sir!

[Y E O N N E]

Tahawus, on 28 September 2016 - 07:46 PM, said:

A sizable majority of the mechs are reasonably balanced. For example 42 of the 67 chassis evaluated have residuals between 0.1 and -0.1. On each end of the spectrum there are 5-6 mechs that I'd assert are extreme values and should be considered for adjustment by PGI.

You need to be extremely careful with this statement. Because your data is investigating performance per ton, saying that the outliers at the end need to be adjusted implies that performance should be worse for lighter 'Mechs. I do not think that the Locust, for example, needs adjusting down; it's good relative to its weight but it's not strictly good overall.

[Tahawus]

That's exactly the reason I said in the post above that this analysis should not be the sole rationale for rebalancing.

That said, I, as a fairly frequent locust pilot think that they are not fairly balanced for their weight. I can get away with stupid crap mulitiple times in a match that would get any other light that I have destroyed on the first pass.

This does need to be balanced against their remaining viable. I don't envy the person trying to split that hair.

[Wecx]

Results are pretty much what i expected to.

Not surprised the KDK is on top (I own one and most of you guys do as well)
Not surprised the Hunchie IIC is up there or the Nova (The hunchie has high mounts, and the Nova can wreck faces)
Not surprised the Locust is up there (The Locust is pretty fast and if you chase it you will get wrecked by a heavy mech, but if you ignore it you will get wrecked so you just pray you have a Light hunter on your team)

Not surprised about the Cheetah (If i had to pick one light out of all the lights in the game it would be the Cheetah it is the most well rounded, you can SPulse, you can ERMED, or Large Pulse, you can even use SRMs plus you have ECM)

Not surprised with the Battlemaster (High mounts, I own one and it can poke for days)

Victor on the bottom
P,Hawk on the bottom
Myst Lynx on the bottom

All really not surprising data to anybody who has played these mechs, It is nice to see a little data to tell us what we already knew.

Why is the Black Knight so low? Well the Warhammer came out, and it can spam the same build, but with Higher Mounts.

My Mauler usually doesn't do quite as well as my Battlemaster, well because the mauler has Lower Mounts, and the Engine Cap of course!
Ebon Jag is a tad better than the Timberwolf, Well guess what it has more Pod Space and hardpoints.

Basically, High Mounts and High Pod Space = Better Mech. (Engine cap has an impact as well)

You can either hit targets easier or fire more guns.
(PS, Please add LIght XL for IS)

Edited by Wecx, 29 September 2016 - 09:02 PM.

[Aggravated Assault Mech]

Requiemking, on 10 September 2016 - 08:40 PM, said:

Ok, but nerfing an entire chassis is not necessarily the best option. For example, it is a well known fact that the Kodiak 3 is OP. However, this is not reflected on the leaderboards. As such, your data, while valuable, is incomplete, and balancing around incomplete data creates a headache of problems when mechs that are balanced get hit with an unnecessary nerf. Case in point, the Jenner. The rescale nerfed the Oxide, just like everyone wanted, but the other variants, which were already balanced and didn't need to be nerfed, got nerfed anyways. Whats more, they got nothing in compensation, which is why the most of the lights in-game are Arctic Cheetahs and Locusts.

I don't think anyone is contending that this info should be used for balancing. Rather it's valuable in putting data behind the common unsupported (or anecdotal) complaint that Clan mechs are substantially better than IS mechs. No, IS pilots aren't crazy/bad, Clan mechs really are just better on average.

I think it stands to reason that dividing it up by variant would only emphasize the disparity between IS and Clan as well. I would expect Omnimechs to shift upwards and the IS presence at the bottom of the chart to be heavily padded by the tremendous quantity of trash variants that are basically unplayed except at tier 4/5.

Edited by vnlk65n, 30 September 2016 - 01:24 AM.

[Precentor Ward]

Boogie138, on 21 August 2016 - 11:48 AM, said:

How did you account for pilot skill?
-giggles-

With math..


























































































-Giggles-

[MechWarrior5152251]

Performance "per ton" is not relevant. At least not in quick play as we are not tonnage restricted.

Of course this does illustrate how OP the Kodiak is at all tonnages.

[Y E O N N E]

MechWarrior5152251, on 16 October 2016 - 03:45 PM, said:

Performance "per ton" is not relevant. At least not in quick play as we are not tonnage restricted.

Of course this does illustrate how OP the Kodiak is at all tonnages.

I would also point out that you can't have one game mode that relies on tonnage to balance 'Mechs and one where all 'Mechs are considered equal. You cannot possibly balance that.

*glaring at FW*

Edited by Yeonne Greene, 16 October 2016 - 09:14 PM.

[Lightfoot]

I don't think you can extract an actual 'best mech' comparison between Clan Tech and Inner Sphere tech because MWO doesn't score for accuracy. It scores for Damage. Clan weapons were balanced to be less accurate than Inner Sphere weapons so although they do more damage, that damage is spread across more mech sections thus less effective.

This is evidenced by the placement in the list of some of the top killer, most dangerous, mechs in the game being located midway in the pack instead of at the top.

Now when MWO adds the stat, 'Kill Most Accurate Damage Dealt', you might get a more realistic comparison of the true Best Mechs in the Game. MWO just tracks KMDD, but if less damage is required to get the KMDD it's a negative scoring effect.

Edited by Lightfoot, 16 October 2016 - 10:06 PM.