262 replies to this topic

[abloobloo]

You often see pugs on these forums cry foul of premades. They claim that premades ruin the game, and that pugs are constantly being wiped off the map, round after round. Eventually, after hearing the anti-premade mantra over and over again, I thought to myself "is this really the case?" Surely, if it were true that pugs were systematically losing matches, that would suggest a deeply flawed matchmaker that was systematically pitting pugs against premades more than against other pugs.

So I started a thread on this forum asking anybody who has never dropped as a premade to post a screenshot of their stats. Lots of people responded by posting their stats, and in total, I had gathered 31 data points (sorry, if you typed your stats, they weren't included, l2 read instructions ).

Cool.

Hypothesis
I went into this with the hypothesis that the win/loss ratio of pugs would cluster at around 1:1 indicating that they are, on average, giving as good as they get.

Findings




In this chart we can clearly see the distribution of both kill/death ratio and win/loss ratio. As I predicted, the win/loss ratio appears distributed around the 1 line, indicating a near 1:1 win/loss ratio for pugs. The average win/loss ratio is 0.92:1 with a standard deviation of 0.31. This is very close to 1 and represents a strong rebuke to pugs who claim they are constantly being steamrolled due to the fact that they are pugging. Some pugs are losing more than they are winning, some are losing and winning about the same number of matches, and some pugs are winning more than losing. This data suggests that a total steamroll of all pugs always, is unlikely.

On the other hand, win loss ratios are comparatively high at 1.49:1 with a standard deviation of 1.11. My guess is that kill/death ratios are higher because this stat is more dependent on personal skill, and mech loadout. If you're a good player with a good mech, on a bad team, you might well take out 2 or more enemy mechs before losing.

With my hypothesis proven correct, I started to look at other things I could do with the data so I tried to correlate some of the stats.



What I found was that the only thing that really correlated strongly were kill/death ratio and win/loss ratio. However, I was disappointed to see no strong relationship between player performance and the amount of MC in their accounts.



So, I decided to split up the data into one group of participants who had MC in their accounts and one group that didn't have MC and compare their average performance. In this chart, we can see that people with MC in their accounts experienced a 51% improvement in kill/death ratio over non-paying pugs. However, in terms of win/loss ratio, the numbers did show a slight advantage for paying pugs, but after running a t test, the p value exceeded 0.05, so take that result with a grain of salt.

Conclusion
Based on the sample, the pug player experience is not uniformly bad; as a group, they are very close to a 1:1 win/loss ratio and a 1.49:1 kill/death ratio. While personal experience for pugs varies by individual, it is not accurate to state that all pugs are experiencing catastrophic losses. If you are constantly being steamrolled as a pug, you should be looking towards personal improvement, rather than looking towards the developers to elevate pugs as a group, because as a group, they're already very close to a 1:1 win/loss ratio.

Furthermore, if you are a pug and are looking to improve your win/loss ratio, there is a relationship between having MC in your account and your kill/death ratio, there is also a relationship between between kill/death ratio and win/loss ratio. And before anyone says it, yes, I know correlation is not causation. I make no claims of the existence of causal links here.

Raw data

If anyone more experienced with statistics than I am would like to poke around the raw data, here's the excel file.

https://www.dropbox....fw8e/TXdkfNMzOP


If the devs would like to weigh in with the actual pug performance stats, I would be really interested to see how well it lines up with my small sample.

[Serapth]

Study is flawed on the nature of epeen waving.

[Thirdstar]

Study is flawed because he has 31 data points. 31! Yeeesh

But sure lets have another flame war because OP think 31 is a reasonable sample.

Edited by Thirdstar, 23 January 2013 - 07:59 PM.

[Ghogiel]

Thirdstar, on 23 January 2013 - 07:57 PM, said:

Study is flawed because he has 31 data points. 31! Yeeesh

out of people who post and read the forum.

[Murku]

People happy with their stats much more likely to contribute them.
Pugstomping is time sensitive, occurring at US peak playing hours.
MC in account? I had 5000 odd MC to start, but have spend it all. Stats just show current.

While I appreciate the study you have attempted your results seem to tell you exactly what you originally expected to here. Thus, not very objective.

Edited by Murku, 23 January 2013 - 08:02 PM.

[Rifter]

Sorry to break this to you but you are going to need 2000+ samples to get reliable stats.

[Mr 144]

While obviously you've put a lot of work into this, I still don't think there's enough valid data. The main problem is your original thread's stat entry requirements were 100% Solo-Pug. Very few people on these forums do this. Almost everyone on these forums will at least run in a duo to 4-man, even using soley the in-game functions every once in a while. For example, I am no longer a Pugger, because even after 1200 matches doing so, I've now spent another 300 matches in TS. I also would not have qualified in my first 1200 matches, as I did duo/group every once in a blue moon for objectivity. This begs the question of the honesty of the data gathered. I'm not referring to blatant stat fraud, but rather the "I mostly Pug' crowd who entered under false pretenses. I do appreciate it's probablly the best data you could gather atm though.

And people exagerate on the interwebz?

Mr 144

Edited by Mr 144, 23 January 2013 - 08:09 PM.

[Thirdstar]

Murku, on 23 January 2013 - 08:01 PM, said:

People happy with their stats much more likely to contribute them.
Pugstomping is time sensitive, occurring at US peak playing hours.
MC in account? I had 5000 odd MC to start, but have spend it all. Stats just show current.

While I appreciate the study you have attempted your results seem to tell you exactly what you originally expected to here. Thus, not very objective.

OP doesn't understand that correlation =/= causation either.

[abloobloo]

abloobloo, on 23 January 2013 - 07:46 PM, said:

Furthermore, if you are a pug and are looking to improve your win/loss ratio, there is a relationship between having MC in your account and your kill/death ratio, there is also a relationship between between kill/death ratio and win/loss ratio. And before anyone says it, yes, I know correlation is not causation. I make no claims of the existence of causal links here.

Thirdstar, on 23 January 2013 - 08:03 PM, said:

OP doesn't understand that correlation =/= causation either.

Yes I do.

[MischiefSC]

Self-selected metrics are always dubious, most the pugs probably never even see the forums. I suspect premade stats are closer to 1.5 or 2 to 1 for wins and at least that for K/D. A comparative analysis of premade vs pug stats would be more useful but again, hard to to get a good sample with self selected participants.

However, I find your data pretty relevant. I only pug. I don't do teamspeak. However, if a premade shows up (they'll be the 4 guys who all run off at a sprint in the same direction without any chat activity) I fall in behind them and try to help. Otherwise I try to motivate pug groups into some general tactics.

Almost universally though you can watch the map a bit and tell in the first 30 seconds how bad it's going to be. If you've got 2 or more people off on their own, you're screwed. Try to get a kill or two in before you all get split up and slaughtered.

I'd say that the big issue is that premades and better communication compensates for average skills and facilitates boating - you can make your mech to work in tandem with another design and know they'll be there. Pug success depends a lot on your loadouts ability to adapt.

I suspect that if you take premades and pug-drop them you'd find metrics falling very much into line. Communication and coordination however simply raises average mein performance - it's easier to cover a teammates skill deficiencies on teamspeak. Whoever is best at identifying targets lends that skill to everyone else, whoever has the tactical skill advantage lends the skill, whoever is boating one design or another finds their inherent gaps filled by teammates who know and can adapt.

In terms of pure dead on killing skills though, one on one, I'd guess the best puggers would put the best premades through their paces.

I don't suppose you want to do a similar sampling of 30 or 40 premade players?

[White Bear 84]

Interesting.

Stats should be taken with a grain of salt, you could look at the MC chart and make the assumption that players with MC are solidifying the 'P2W' mantra, but what if they are just more experienced players - how many MC holders are founders or have played a MW game before? What is the age group of the MC/non MC holders. You need more data to correlate - you cant build a car unless you have all the right parts

You can say that pugs have a roughly average 1-1 win/lose ratio but that in meaningless in its own context. The question you raise is about PUGSTOMPING - matches where pugs get steamrolled by premades, but there is no data here that answers that question.

What would make the study more valid is if you looked at each of the players and provided a correlation between the number of times they won/lost and the number of mechs destroyed - take into account capping win/losses, not to mention disconnects, the builds that they have run, previous experience, the composition of their teams e.g. are they playing with other premades when they win/lose, what is the actual proportion of premades in each game, how many people are communicating on TS, ping, tonnage and firepower of each team etc.

For all you know, the players with the higher KDR could have dropped against fewer premades than those who have a lower KDR. A correlation of data without facts to back it up is meaningless.

Regardless, i dont want to flame you.. ..these are still interesting statistics to see and kudos on taking the initiative - i would suggest though if you want to prove a hypothesis you substantiate the results with more concrete data. :/

Edited by White Bear 84, 23 January 2013 - 08:14 PM.

[Kaspirikay]

inb4 second page of threadnaught

inb4 second page of threadnaught

inb4 second page of threadnaught

[Sigmund Sandoval]

Although impressive, I would be interested to see a similar study with a higher sample rate. I do believe that there is some very relevant data here though.

[Thirdstar]

abloobloo, on 23 January 2013 - 08:07 PM, said:

Yes I do.

If you do understand then you would have never said that there's a relationship between having MC in your account and K/D. Because there's no evidence of a relationship at all. Hence, no you still DO not understand that correlation =/= causation.

If my old Stat prof had read your post he would had an apoplectic fit.

[abloobloo]

Thirdstar, on 23 January 2013 - 08:12 PM, said:

If you do understand then you would have never said that there's a relationship between having MC in your account and K/D. Because there's no evidence of a relationship at all. Hence, no you still DO not understand that correlation =/= causation.

If my old Stat prof had read your post he would had an apoplectic fit.

There is a relationship, just not a causal one. I made no claim that one causes the other. I posted it because I thought people might be interested in it.

People in my sample who happened to have MC in their accounts on average had higher k/d ratios than the people without.

Edited by abloobloo, 23 January 2013 - 08:18 PM.

[Thirdstar]

abloobloo, on 23 January 2013 - 08:15 PM, said:

There is a relationship, just not a causal one. I made no claim that one causes the other. I posted it because I thought people might be interested in it.

People in my sample who happened to have MC in their accounts on average had higher k/d ratios than the people without.

This does not suggest any kind of relationship at all. So, why did you suggest there was one despite lack of evidence to support your assertion?

I dislike people who have an Agenda but won't come out and say what it is.

Edited by Thirdstar, 23 January 2013 - 08:22 PM.

[Ryvucz]

That's like opening one bag of M&Ms, counting how many are what color, and throwing everything with just that sample bag.

"But don't worry, instead of blaming the bag of M&Ms for not having more of the color you want, look at yourself and change."

[Feetwet]

He'll some of the samples look a little dubious. I would throw out the first one as hoakie to begin with. While I agree with you that 1:1 isn't that bad while pugging, I doubt most pugs are there. As 100% pluggers go I feel like I'm doing ok but I am still at a win rate of 42%. I drive a hbk and am steadily improving my k/d, but it's brutal out there.

S

[abloobloo]

Thirdstar, on 23 January 2013 - 08:21 PM, said:

I dislike people who have an Agenda but won't come out and say what it is.

We call them hypotheses.

White Bear 84, on 23 January 2013 - 08:09 PM, said:

For all you know, the players with the higher KDR could have dropped against fewer premades than those who have a lower KDR. A correlation of data without facts to back it up is meaningless.

Regardless, i dont want to flame you.. ..these are still interesting statistics to see and kudos on taking the initiative - i would suggest though if you want to prove a hypothesis you substantiate the results with more concrete data. :/

Obviously there are flaws with the sample size and sampling bias. After all, I had to work with what I could get. What was most interesting to me was the clear trend of win/loss scores clustering around the 1.0 line. I think there's definitely something in that.

[Thirdstar]

abloobloo, on 23 January 2013 - 07:46 PM, said:

As I predicted, the win/loss ratio appears distributed around the 1 line, indicating a near 1:1 win/loss ratio for pugs.

You clearly have no idea how to read charts either.22 of your 31 samples are AT or BELOW the 1 line. How then is the average W/L magically 1:1?

Protip: Don't start with a biased hypothesis.

Clustering sounds like a weasel word here. Why don't you just say the majority of the sample was below 1:1 W/L, you'd at least be accurate.

Edited by Thirdstar, 23 January 2013 - 08:27 PM.