16 replies to this topic

[Rauchsauger]

Hi,

I have a quick question regarding "Tweaks" to MWO based on the 90% win ration Clan has over IS.

How did PGI measure that? There are so many variables.
Were those games set up with known players?
If not:

How many matches did they use for this evaluation?
Given the matches the matchmaker does create I highly doubt that they measured equally skilled teams facing each other.

I assume most people with clan packages are veterans. From that I take that said veterans would play mostly vs more casual players. (Again I don't know if they chose a select player base for those tests) If that would be the case the results from that "survey" would not be significant.

Does someone have more information on this?

[SgtKinCaiD]

I think you have missed the train.

[DeRazer]

Post nerfs in the recent Clan vs IS games it was down to 73%. Which feels about right.

[Asmosis]

it should be about 50%, so that'd mean more balancing/nerfs needed.

[Modo44]

Things were nerfed from completely silly to silly. Now the issue is whether you keep nerfing to get that 50%, or whether you say **** it and just do different drop sizes for CW. The mixed tech used in public matches negates a lot of the advantages, so once pay to win stops being a thing (i.e. when the Timber Wolf comes out for Cbills), Clan tech may remain OP on purpose.

[Rauchsauger]

But what about the question I asked?

does anybody know if the matches they used to get the data were set up?
Because if the matches were generated by the "matchmaker" the gathered data would not be significant since the
so called matchmaker does not generate evenly matched games + most people playing clan at the time of the test
would most likely be veterans.

So my question remains unanswered - Is the collected data statistically significant or not?
(and please do only answer if you have information on this and you know what statistically significant means)

Thank you in advance

[SgtKinCaiD]

Then you should take a look at the post made by Kiior => https://mwomercs.com...s-with-science/
But it was done before the latest Clan Weapons balance phase and the latest IS vs Clan test (the one with 73%).

[Rauchsauger]

SgtKinCaiD, on 23 September 2014 - 02:18 AM, said:

Then you should take a look at the post made by Kiior => https://mwomercs.com...s-with-science/
But it was done before the latest Clan Weapons balance phase and the latest IS vs Clan test (the one with 73%).

Wow thank you - that is a lot to digest.

(but you do not have any more info on how PGI collected their data (and hopefully cleaned it up) when they set the matchmaler to IS vs Clan?)

[SgtKinCaiD]

The only intel we got about the last IS vs Clan test is that Clan won 73% of the match and there was a average team ELO deviance of 100 in favor of Clan.

[Modo44]

Rauchsauger, on 23 September 2014 - 02:14 AM, said:

does anybody know if the matches they used to get the data were set up?
Because if the matches were generated by the "matchmaker" the gathered data would not be significant since the
so called matchmaker does not generate evenly matched games + most people playing clan at the time of the test
would most likely be veterans.

Yes, it was significant data despite the matchmaker. Manually set up matches are statistically insignificant. Thousands of matches played by everyone is where you get actually useful data.

[Darwins Dog]

They haven't really said one way or the other (as far as I know). They did mention that the latest test ended early because people were complaining about losing too much. There were also widespread reports (though very hard/impossible to confirm) that people were throwing matches intentionally.

I would hope that they are at least cleaning it up a little to remove anomalies (remove DCs, Teamkills, etc.), but in the grand scheme of things, they don't really want their data to be too clean. They are concerned with overall player experience, not the perfect match. This is my bias as an ecologist showing, but it's like the difference between field and lab studies. In the lab one can control for all sorts of variables (tonnage difference, DCs, Elo, 4x3, builds, etc.), and isolate only the relevant treatment/response (tech type/win ratio). In a field experiment one manipulates the desired treatment (no mixed tech), but is forced to let the other variables go where they may. The tests that they have run are closer simulations of how the general populace will perform.

Of course, the test also depends on the question. If the only matches with fixed tech are going to be organized teams (as per CW description), then the tests should be done with organized teams. I guess what PGI has done is more comparable to a pilot study then a conclusive experiment.

[Rauchsauger]

Modo44, on 23 September 2014 - 04:08 AM, said:

Yes, it was significant data despite the matchmaker. Manually set up matches are statistically insignificant. Thousands of matches played by everyone is where you get actually useful data.

ouch. That depends on what you want to measure. If you want to measure the balance of mechs and you just set your matchmaker to IS vs Clan then you are not measuring the mechs...
You also have to get rid of player skill, team skill for example. Given the time when they did the test and the duration of it (from what I could gather) They pitched more veterans (clan) vs new players (IS). The data from Kiior suggests the same.

In other words those thousands of matches were ALL lobsided by the influences stated above so the data is not statistically significant if it is not cleaned up by for example a control group (which is a manually setup group of course) so please Modo44 I just want to know if anyone KNOWS how PGI does their research / if said research was done correctly.

Edited by Rauchsauger, 23 September 2014 - 06:20 AM.

[Pariah Devalis]

As a function of the Clan teams having an Elo advantage between 100-200, Russ stated they are aiming at a 65-35 or 60-40 win loss for clan IS as a 50/50 win loss would overnerf the Clan tech. Pilot skill OP.

Edited by Pariah Devalis, 23 September 2014 - 06:28 AM.

[Rauchsauger]

Darwins Dog, on 23 September 2014 - 05:06 AM, said:

[...]
I would hope that they are at least cleaning it up a little to remove anomalies (remove DCs, Teamkills, etc.), but in the grand scheme of things, they don't really want their data to be too clean. They are concerned with overall player experience, not the perfect match.
[...]

I think it is perfectly OK if they have an eye on overall player experience but that test (again - as far as I know) also did not measure the overall experience since the time they made the test carried special circumstances. (Like I stated in the above post to Modo44)

Also if you want to achieve that changes to mechs are not what you need to do - but changes to gameplay like handicap or a functioning matchmaker. Or other balancing factors like tonnage limits instead of number limits on games and / or economy back in game.

Edited by Rauchsauger, 23 September 2014 - 06:33 AM.

[Modo44]

Rauchsauger, on 23 September 2014 - 06:19 AM, said:

In other words those thousands of matches were ALL lobsided by the influences

Of normal matchmaking, without special balancing against Clan mechs, therefore effectively measuring how strong Clan tech really is. But hey, why accept facts.

[Kjudoon]

Individual mechs dont need nerfs. Mixing Clan/IS needs to be ended, and see if it needs to be 10v12 only.

This should remain the case till Clan weaponry and equipment is mountable on IS mechs (as will ultimately happen over the course of time in any war).

[DEMAX51]

Per Russ' tweets:

- They collected data on something like 260 matches.

- The Clans won 73% of the time.

- The Clan teams had a generally higher Elo (so yes, they ARE taking player skill into account here)

- Russ feels that Clan's may not need any more weapons nerfs, but he hopes that the quirks they're about to give to IS 'Mechs will level the playing field, and he will think about adding penalties to Clan 'Mechs with XL engines if one of their side-torsos is destroyed (such as lower heat dissipation, or, possibly, a reduction in speed/mobility).

But yeah, man. If you think their statisticians don't know that they have to account for player skill when running these tests, you're just being foolish.

Edited by DEMAX51, 23 September 2014 - 08:30 AM.