89 replies to this topic

[White Bear 84]

Interesting job - alot of work would have gone into this; i think there is more you can do to make the statistics more significant.

First of all looking at the first post there is no explanation on the task completed, what was recorded and when? It would be good to turn the tables into charts or something visual - this means alot more to many people than a set of numbers does

Secondly, i cannot see a reference to the number of sample matches - this is highly significant given that a high occurance on a low number of matches is less significant than a low occurance on a high number of matches.

Thirdly - A limited selection of players is significantly biased, these 13 players have a degree of control over which mechs appear in game adding bias to the sample; was their selection excluded from the end results?

All just thoughts.

The mech selection is pretty interesting though - apparently people ARE buying the heavy metal! Also interesting to see two trial mechs so high up - could be suggestive of casual players or new players.. ..Not suprising that the 3L and DDC are so high up!!

[Arcturious]

Just to point out that using the data from the weekend as a judge of mech diversity is not going to be accurate, thanks to the bonuses that were in place for founders mechs and heroes, so it will be slightly skewed for those.

It will however still be accurate for matchmaking, ECM etc which is great.

It will also be limited to data from the elo brackets of the people submitting the data. It's quite possible that in their averaged elo bracket, heavies are run more often. Perhaps in lower / higher brackets the results would be different.

What I would love to see is some sort of community elo investigation next. The six degrees of mech segregation or something.

You simply start parsing the same data, and seeing who shows up in which match with people consistently. Then look for commonalities in other matches, sort by win / loss and see if you get a very rough elo chart. Would need vast amounts of data to do it though I think.

Thanks for the work you've put in on this, numbers are always interesting!

[Noob Weapons]

Good stuff. +1 to more in the future.

[Ari Dian]

There is only one bad part on this statistics.
The fact that these should have come from PGI. Like the walk/death maps they were presenting some time ago.

Some well presented data like these could tell you more than endless words. And, like in this case with the matchmaking, show that its really fine and working. I was expecting something informative like this from the devs.

Great job on it. Look good, is helping and give some info.

[Taemien]

This sort of confirms an argument I've been making for a while. ELO does work and the matchmaker works great. But it needs the numbers to make it work. This weekend with many playing at once allows the matchmaker to do the job it is intended. But yet we still see complaints about it. I don't think players are pulling experiences out of their back end, so I assume what they say is true, though most of the time it is due to non-peak times. Unfortunately for them, no amount of coding to the matchmaker will fix their issues. Simply just need more players during those off peak times.

Course that isn't an issue that is unique to MWO, all online games have this issue.

[Jace Lancer]

Pretty cool info

[Matroid]

Hammerborn,

nice job at collecting data. can you tell if i was the pilot of the TBT-7K? there were at least 3 reported instances, and i had bought mine on that Friday

[Belorion]

It would be neat to see this kind of data without the special event. Perhaps a Tuesday after the weekend event ends till Friday right before the next one starts.

It would be nice if we could log games. That would make this kind of thing a lot easier.

[Corwin Vickers]

The ECM "not making a difference" might be heavily influenced by the fact that there are barely any LRMs out there anymore.

[Hauser]

Awesome work hammer. You're a crazy **** for typing it all out!

I'll try to keep an eye out for the next collection.

White Bear 84, on 10 April 2013 - 10:51 PM, said:

Thirdly - A limited selection of players is significantly biased, these 13 players have a degree of control over which mechs appear in game adding bias to the sample; was their selection excluded from the end results?

Good question!

The observers mechs have been excluded from the count. You can see this in the raw data sets. 7 mechs are counted for the friendly team. 8 for the enemy team. The observers mech is included in the weight counting though.

The original data is still up so you should be able to find that quickly. Then you can use that to see if the percentages change significantly when playing a different chassis (they should slightly, because you get matched against at least one mech near your own weight).

Arcturious, on 10 April 2013 - 11:08 PM, said:

What I would love to see is some sort of community elo investigation next. The six degrees of mech segregation or something.

You simply start parsing the same data, and seeing who shows up in which match with people consistently. Then look for commonalities in other matches, sort by win / loss and see if you get a very rough elo chart. Would need vast amounts of data to do it though I think.

I'd be nice if we'd have some logs. Mech variants are okay. Typing out names is going to be pain.

Edited by Hauser, 11 April 2013 - 05:14 AM.

[hammerreborn]

Oh man this is going to be awful typing from my phone:

Lets begin!

White Bear 84, on 10 April 2013 - 10:51 PM, said:

Interesting job - alot of work would have gone into this; i think there is more you can do to make the statistics more significant.

First of all looking at the first post there is no explanation on the task completed, what was recorded and when? It would be good to turn the tables into charts or something visual - this means alot more to many people than a set of numbers does

Secondly, i cannot see a reference to the number of sample matches - this is highly significant given that a high occurance on a low number of matches is less significant than a low occurance on a high number of matches.

Thirdly - A limited selection of players is significantly biased, these 13 players have a degree of control over which mechs appear in game adding bias to the sample; was their selection excluded from the end results?

All just thoughts.

The mech selection is pretty interesting though - apparently people ARE buying the heavy metal! Also interesting to see two trial mechs so high up - could be suggestive of casual players or new players.. ..Not suprising that the 3L and DDC are so high up!!

All recordings were done this past weekend during the 3050 weekend event. Each set (A-O) are ten rounds totaling 150 mechs, 15 per round as the user is excluded. Rules were set up on the linked page to the raw data, for instance, all had to be playing solo, must be playing assault mode, and no disconnects allowed in the match as it skews the winner completely. All the raw data is in the following posts, with links to the submitted screenshots used to provide the data.

If you have any concerns about it it's all right there. I'm completely transparent when I do these.

Arcturious, on 10 April 2013 - 11:08 PM, said:

Just to point out that using the data from the weekend as a judge of mech diversity is not going to be accurate, thanks to the bonuses that were in place for founders mechs and heroes, so it will be slightly skewed for those.

It will however still be accurate for matchmaking, ECM etc which is great.

It will also be limited to data from the elo brackets of the people submitting the data. It's quite possible that in their averaged elo bracket, heavies are run more often. Perhaps in lower / higher brackets the results would be different.

What I would love to see is some sort of community elo investigation next. The six degrees of mech segregation or something.

You simply start parsing the same data, and seeing who shows up in which match with people consistently. Then look for commonalities in other matches, sort by win / loss and see if you get a very rough elo chart. Would need vast amounts of data to do it though I think.

Thanks for the work you've put in on this, numbers are always interesting!

I actually told all participants to make sure the same people are not popping in every match, and like above, all their raw data is available. There were some humorous overlaps. For instance, aniviron fought hauser in one of his data sets or vice versa, though the other did not have that match in their own data.

With 2250 mechs tabulated though, a few repeats shouldn't effect the results too much, aside from maybe the bottom feeder mechs.

Ari Dian, on 10 April 2013 - 11:56 PM, said:

There is only one bad part on this statistics.
The fact that these should have come from PGI. Like the walk/death maps they were presenting some time ago.

Some well presented data like these could tell you more than endless words. And, like in this case with the matchmaking, show that its really fine and working. I was expecting something informative like this from the devs.

Great job on it. Look good, is helping and give some info.

But then i would be out of a job ;(

Hadros, on 11 April 2013 - 04:03 AM, said:

Hammerborn,

nice job at collecting data. can you tell if i was the pilot of the TBT-7K? there were at least 3 reported instances, and i had bought mine on that Friday

I can check when I get to a real computer, but the quickest way you can do it is to find the data set listed at the top where a K was seen, opening the corresponding chart and screenshot set below, and matching the rounds up.

EDIT: You are not.

Corwin Vickers, on 11 April 2013 - 04:42 AM, said:

The ECM "not making a difference" might be heavily influenced by the fact that there are barely any LRMs out there anymore.

Even when LRMs were prevalent it had the same effect (see round 2: pretty baby patch data). And since most of the arguments are about making d-DCs ninjas that are completely invisible, it shows that's just not the case.

Edited by hammerreborn, 11 April 2013 - 07:27 AM.

[hammerreborn]

Taemien, on 11 April 2013 - 01:01 AM, said:

This sort of confirms an argument I've been making for a while. ELO does work and the matchmaker works great. But it needs the numbers to make it work. This weekend with many playing at once allows the matchmaker to do the job it is intended. But yet we still see complaints about it. I don't think players are pulling experiences out of their back end, so I assume what they say is true, though most of the time it is due to non-peak times. Unfortunately for them, no amount of coding to the matchmaker will fix their issues. Simply just need more players during those off peak times.

Course that isn't an issue that is unique to MWO, all online games have this issue.

The problem is looks can be deceiving. When I was doing this I looked at some matches and I'm like wow, I went against 2 ddcs and D, and my team had 2 stalkers, wtf

Then after I type in all of the mechs I see the difference was actually only 15 tons because we also had 4 cataphracts while they had 3 centurions.

Looks are very deceiving when it comes to team weight, since you're typically exclusively looking at the assault counts.

There are some legitimate comings though. A few rounds broke a 100 ton difference, the greatest being 135. The heavier team lost, but if it went the other way, it can change your perceptions quickly.

Edit: it also depends on what you think good matchmaking is. Some people might find that a 30 ton difference is way too much.

Edited by hammerreborn, 11 April 2013 - 06:25 AM.

[MaddMaxx]

Thontor, on 11 April 2013 - 06:36 AM, said:

I would love to see this kind of data collected at a time when something like the '3050 Weekend' isn't skewing the numbers towards Founder and a Hero Mechs.

Just as a point of interest. The fact that it was a Founders/Hero based Bonus weekend, I ran my Founders (a Full Set) rather sparingly really. The fact that I could make more C-bills in a Founder was heavily outweighed by the fact that in order to best serve the cause, my ride(s) of choice are not my Founders Mechs.

P.S. When I did drive one of my Founders, it was in the same weight class as what I normally drive. So that speaks to the lack of change the Bonus would make to the MM and its match ups in the games...


P.S.S. Great job OP. Will be more helpful next time. (I sorta promise)

Edited by MaddMaxx, 11 April 2013 - 07:06 AM.

[hammerreborn]

Thontor, on 11 April 2013 - 06:36 AM, said:

I would love to see this kind of data collected at a time when something like the '3050 Weekend' isn't skewing the numbers towards Founder and a Hero Mechs.

Look at my round 2 set.

Also, generally these days something is always skewing results, with heroes released the first week of the month and a mech the third week (after the heavy metal results I see no reason they wouldn't continue this trend). People are always going to gravitate towards the new thing so other than a time with no resent content added the results will always be skewed in some way or another. Even maps and weapons changes have an effect, look how much the a1 dropped between this set and the last.

[hammerreborn]

Thontor, on 11 April 2013 - 07:17 AM, said:

You should probably combine multiple sets of data, over an extended period of time, to account for those biases towards new mechs or bonuses to rewards

You could keep them separate, to show how mech diversity is changing over time, and how these biases affect mech diversity. and then have a combined data set to show overall trends. Making a note for each time period as to what the new mech is, or any buffs or nerfs that might have affected people's mech selection.

All this data was collected over the 3050 weekend right? I'm just curious to see longer term trends.

The problem is mechs are added periodically, so a long term trend is useless, because all the older mechs will have a strong bias.

And yes all of the data was collected over the 3050 weekend.

Also, you can see trends from each set I've done. I did one in december, one in march, and now one in april.

The biggest and easiest trend to see is the extinction of lights, who were dominant in december with the release of the ECM raven, to by far the lowest played class overall now as more and more people can actually hit them. They are stable around 10%, but I think the ballistic hit they are about to take will drop them to 7%ish overall.

Edited by hammerreborn, 11 April 2013 - 07:24 AM.

[hammerreborn]

Thontor: I can do that with my 3 sets when I get home most likely.

Annnnddddd the point when you realize you messed up. So while I was typing up and explanation on how the data works I wrote "I excluded the users mech in everything but the weight" and realized OH **** I didn't include it in the ECM numbers.

Then I started flipping through and realized only Rigatoni used an ECM mech nearly exclusively (the 5D), and shouldn't take me too long to correct.


EDIT: AND HUZZUH! It made no difference. Still ends 6-4 as -1s become 0s as much as 0s become 1s.

EDIT2: Looks like his second set does not fare so well, and ends 6-5 instead of 6-3, which does chance things a bit. I'm going to go over these numbers with a fne tooth comb when I get home.

Edited by hammerreborn, 11 April 2013 - 08:09 AM.

[hammerreborn]

Thontor, on 11 April 2013 - 08:18 AM, said:

One thing I just thought of. I think the comparison from data set to data set only works if the data sets are of equal size... So use the same X number of matches for each one, otherwise your Variant A numbers could be higher in Data Set 1 simply by Data Set 1 being larger.

You would use the overall percentage which would eliminate all those concerns (so you would see how the overall % of d-DCs increased over time rather than just the raw numbers)

[General Taskeen]

ShadowDrake05, on 10 April 2013 - 08:49 PM, said:

Edit: Also, ECM win 54%? Big deal. Certainly not the "OMG ECM BREAKS THE GAME" argument it should be. Best be careful with that statistic

Regarding this specifically, all I see it as Stockholm Syndrome's patting themselves on the back. ECM isn't broken, it does what PGI programmed it to do, and people don't agree with how they programmed it (in fact, quite a LOT).

People specifically have THINK TANKED better ways for ECM to be implemented to be a better IW tool with passive abilities. The best argument for a F2P game is that you don't implement devices/weapons/etc. that require gear to be equipped to counter one such device. It is a poorly thought out mechanic.

Specifically ECM in MW:LL is the only Mech Warrior game to refine MW3/4 ECM incarnation, make it better than those two games, but refine it, and have it function in a better format.

The are two differing camps with PGI ECM, those that like PGI's programming, and those that prefer something 'like' previous ECM in Mech games that were a closer representation to the TT ECM.

There are other issues regarding these tests, since it doesn't account for a missile fix (lack of weapons used), the average level of the ELO Matches reported (since ELO is hidden and really only PGI has that data), and at least half or more of weapons that are the least used, at this time, are not being taken into account.

The tests are good for determining the diversity of Mechs and Variants used, but coorelating data is more difficult without knowing certain factors.

[Misoski]

How did the ECM players play? randomly die alone? taken a single enemy straggler and killed/weakened him alone? Great stealthy caps or help hide part of team for flanks?

[hammerreborn]

General Taskeen, on 11 April 2013 - 09:48 AM, said:

Regarding this specifically, all I see it as Stockholm Syndrome's patting themselves on the back. ECM isn't broken, it does what PGI programmed it to do, and people don't agree with how they programmed it (in fact, quite a LOT).

People specifically have THINK TANKED better ways for ECM to be implemented to be a better IW tool with passive abilities. The best argument for a F2P game is that you don't implement devices/weapons/etc. that require gear to be equipped to counter one such device. It is a poorly thought out mechanic.

Specifically ECM in MW:LL is the only Mech Warrior game to refine MW3/4 ECM incarnation, make it better than those two games, but refine it, and have it function in a better format.

The are two differing camps with PGI ECM, those that like PGI's programming, and those that prefer something 'like' previous ECM in Mech games that were a closer representation to the TT ECM.

There are other issues regarding these tests, since it doesn't account for a missile fix (lack of weapons used), the average level of the ELO Matches reported (since ELO is hidden and really only PGI has that data), and at least half or more of weapons that are the least used, at this time, are not being taken into account.

The tests are good for determining the diversity of Mechs and Variants used, but coorelating data is more difficult without knowing certain factors.

Indirectly it does. The missile fix directly influences which mechs are played. You see a rise in poptarts which aren't reliant on missiles and a fall of boaters such as the A1.

And if you have a way to determine the loadout of every mech seen in a match without targetting and screenshotting every user, then dying immediately to spectate everyone on your team, by all means let me know.

The average ELO doesn't really matter either, and I'm not sure why you would want to know that.

Misoski, on 11 April 2013 - 10:02 AM, said:

How did the ECM players play? randomly die alone? taken a single enemy straggler and killed/weakened him alone? Great stealthy caps or help hide part of team for flanks?

Look at the screenshots and make your assumptions from there. Should be pretty easy.