85 replies to this topic

[MischiefSC]

It is reasonable to say there will be a bit of skewing based on better players picking better mechs and bad players picking bad mechs but the most you could possibly hope to get by accounting for that is reducing the total point spread.

However in the aggregate you'll have more bad players in good mechs than good players in good mechs. Unless PGI creates and then implements a method of accurately rating player skill on a ladder (like TrueSkill) and then modifies it to account for individual player skill variance between chassis player skill is a not a factor you can effectively account for.

Fortunately large numbers washes out a lot of that impact to begin with.

This is an excellent study on the topic and it simply provides mathematical confirmation of what player choice has already shown - the bulk of top performing mechs are Clan mechs. Because tech isn't balanced and never was and quirks are and always were a mediocre and unpredictable bandaid.

The problem is that at this point PGI has spent *years* leaving tech balance broken from their promise of 1 to 1. There's no surprise here, just good research into more clearly identifying a problem that plays out in the game and has since Clans were released.Quirks are not balance. Never were.

[jaxjace]

Leaderboards dont mean **** because it only takes your 10 best matches, you can do 90 bad matches and it will only look at your 10 best.

[Lukoi Banacek]

Event leaderboards are indeed too narrow of an indicator.

Global however are pretty illuminating. Wish they had much more information however.

[Andi Nagasia]

to see how mechs handle i look at the 60-75th scores,
i feel they give a more realistic view of where mechs and Peoples averages are,

only looking at the top 10 gives a very Narrow almost Elitist(best of the best) view of things
it like only looking at the top 10 in the Military then comparing everyone else in the military with those ten,
not all Olympic swimmers are Michel Phelps, so we cant expect all Olympic swimmers to have Michel Phelps scores,

[InspectorG]

Interesting.

Questions for context of the above info:

Did you account for the fact there are fewer clan mechs than is?

The leader board scores are heavily biased to total damage instead of efficient damage.

The leader board events are basically farming events where the best scores are gotten under the conditions of:
High level pilot +ideal farming conditions (enemy team full of scrubs).

Builds used for said farming may not be the same as builds used for high level play.

Since leaderboards are farming events how is the mech in question handle defense 've good opponents rather than bad punching bagopponents?

Which mechs are generalist vs specialist/niche?

Which mechs are meta-resistant or dependant?

How much does map selection play into a mechs rating?

Thanks for the effort on the info. And sorry about bad spelling/etc. On my mobile.

[Yellonet]

Clans OP. Some of us are not surprised.

[Marquis De Lafayette]

This is excellent work. I looked at the raw data and it looks like you pulled the leaderboards from the Chassis based events (not this current weight based event).

Assuming so, it really is a pretty decent way to mitigate potential pilot skill differences. I know good pilots ran some pretty unloved chassis that they felt they could place really high in (vs. the just the meta, more loved chassis) and thus collect better rewards on those unloved mechs leaderboard (1000 MC!)

I would also be curious to see your thoughts on the chassis based Assaults-only event in particular, as that seemed to have a giant disparity between the Kodiak and everything else.

I ran some basic, rough numbers at the time that event finished, but I didn't double check them as I didn't think I would use them for anything.

According to my notes only 12.9% (97 out of the 750 who ranked on the ten IS assault builds) IS assault pilots scored higher than the 75th ranked score on a Kodiak. The Mauler did the best with 22 out of 75 pilots beating the lowest Kodiak placer. The winner in the Zeus and Victor chassis wouldn't have even placed in the Kodiak rankings and the other 7 Chassis would have had between 18 to 6 scores that beat the 75th placing Kodiak score.

On the Clan side it is still pretty brutal. Only 20% of the other placers in non-Kodiak Clan assaults would have beat the 75th ranked Kodiak score. The Dire Wolf being far and away the best with a respectable 31 pilots beating the lowest placing Kodiak score. All the other Clan assaults had between 15 and 7 pilots whose score would have ranked them on a combined (with Kodiak) board. Similar to the IS assaults (minus the hapless Zeus and Victor).

Based on the data it seems like the Kodiak is in a class by itself.

[Kangarad]

meh just because the quad uac5 IS 100t assault is as good as a quad Cuac5 clan medium does not make clan overpowerd. you guys just need to relax a bit.

/sarcasm btw. because themedium does not hae enought ammo to do more than 900 damage.

Edited by Kangarad, 22 August 2016 - 03:07 PM.

[Chados]

I didn't see the Rifleman or Marauder, and didn't both have leaderboard events?

[Asaru]

Great job Tahawus! This was some really good work and the numbers are very telling.

[El Bandito]

MischiefSC, on 22 August 2016 - 10:03 AM, said:

The problem is that at this point PGI has spent *years* leaving tech balance broken from their promise of 1 to 1. There's no surprise here, just good research into more clearly identifying a problem that plays out in the game and has since Clans were released.Quirks are not balance. Never were.

This is very true. Just take a comparison of IS and Clan version of XL engine, ERPPCs, BAP/NARC/ECM/CASE etc, not to mention FF and Endo on Clan Battlemechs... 1 to 1 balance is far from over.

Edited by El Bandito, 23 August 2016 - 05:34 AM.

[Tahawus]

SchnitzlXS, on 22 August 2016 - 02:59 AM, said:

Hey man,

Let me say, really nice work.

I'll PM you some humble suggestions.

Cheers!

Schnitzl made some great suggestions, some of which had occurred to me, some of which might be able to improve the analysis, and most of which would take a bit more work to do transformations, model results, and then untransform results back to something more easily communicable.

I might get to them at some point because they are good ideas and might resolve some of the technical weaknesses of my approach (non-normality and heteroskedasticity).

InspectorG, on 22 August 2016 - 12:01 PM, said:

Did you account for the fact there are fewer clan mechs than is?

No, in this case, the models are built on the inclusion of all mechs available with in the data set. I have built individual models for clan and IS, they basically replicate the presented results without being able to compare clan vs IS.

InspectorG, on 22 August 2016 - 12:01 PM, said:

Since leaderboards are farming events how is the mech in question handle defense 've good opponents rather than bad punching bagopponents?
Which mechs are generalist vs specialist/niche?
Which mechs are meta-resistant or dependant?
How much does map selection play into a mechs rating?

All questions that I'd love to tackle, but can't given the available data (hint...hint PGI)

Chados, on 22 August 2016 - 03:27 PM, said:

I didn't see the Rifleman or Marauder, and didn't both have leaderboard events?

I'll include all of the heavies when the all heavy leader board comes around. I'm mulling, based on prior feedback whether we have enough comparability of results to keep the viper in and to include the Cyclops based on their individual leader boards.

Dribbles and drabbles...

Edited by Tahawus, 22 August 2016 - 09:02 PM.

[Hit the Deck]

How could kaffeeangst/VictoriaSeymore/S3B be such an anomaly? Is he Kai Allard Liao in disguise?

[zagibu]

Hey, I've written a python scripts that reads the screenshots and outputs the text of the end results screen for Yellonet, who also collected some statistics, but he never got back to me: http://mwomercs.com/...-data-extractor

If you could tell me what resolution your screenshots have, I could try to prepare it for you. It also just prints the text into the console at the moment, but I could do a CSV export or whatever you need.

Edited by zagibu, 22 August 2016 - 11:51 PM.

[Contrex]

At the moment there are 90 IS Mechs in the top 300 of the event. The best part are the 11 IS Assaults out of the top 75!

[MischiefSC]

Actually using the top scores (and as such top players) is a reasonable angle on solving for player skill. The further along the skill curve you get the more consistent the performance results. You'd see a much bigger swing caused by player skill variables the closer you get to average.

Comparing the top scores of top players is actually a more accurate view of the mechs performance. It reduces player skill variable to the lowest possible percentage.

[Constructive Waste of Time]

Hi. If you just look at the best 75 of the assault class of the still ongoing leaderboard, you mostly see Kodiaks and other clan assaults with just one Atlas. Even in live games you mostly see clan mechs. I am a pure IS mech owner and I think we need extra quirks for IS mechs to reequilibrate at least a little in the battlefields. Thanks.

[SchnitzlXS]

Tahawus, on 22 August 2016 - 09:00 PM, said:

Schnitzl made some great suggestions, some of which had occurred to me, some of which might be able to improve the analysis, and most of which would take a bit more work to do transformations, model results, and then untransform results back to something more easily communicable.

I realise that the aim of the analysis is to communicate the results with ease. That is why I proposed a log-log transformation. You mitigate most of the problems with your regression and the model is still straight forward to interpret (coeffs are interpreted in terms of percentage points and not units of measurement). No transformation needed. Just run the regression in the proposed form.

Tahawus, on 22 August 2016 - 09:00 PM, said:

No, in this case, the models are built on the inclusion of all mechs available with in the data set. I have built individual models for clan and IS, they basically replicate the presented results without being able to compare clan vs IS.

With the inclusion of the dummy variable for IS you are un-intentionally accounting for both sides. The difference in number of mechs essentially does not matter much, since the regression analysis is not that much sensitive to the size of subsamples (ofcourse given that micronumerosity and lack of variability in explanatory variables are not a problem).

Hope that helps.

Cheers!

[nehebkau]

Tahawus, on 21 August 2016 - 11:09 AM, said:

If PGI would like me to do more work by giving me greater access to their data, I'm willing and able. I have a significant background in the statistical modeling of behavioral and geographic phenomena.

Also added R-Square's for the regressions

I have advised PGI that they need to hire an experienced statistician. Not just for balance sake but for data mining to detect exploiters. Nice analysis, even my 1/2 century addled mind followed.

[Tahawus]

I'm going to break my rule about posting while on work time, I agree, but I think they could get a lot of benefit from opening API access to diaggregate anonymised match results.

They might also consider a committee of data analysts drawn from the community as a review panel for balance. These folks should be able to demonstrate their educational and professional credentials.