Latest update to the data collection follows
Methodology is still the same as usual: I drop into PUG games with 1 friend on skype. - He helps me count up ECM on the enemy team during the match. In the latest data sets I'm also making an effort to look for disconnects at the start of the game, and not counting the matches where a team loses a player right at the start.
(Out of curiosity I also tried to count TAG systems on each team to get a gage of how often one can count on having a friendly TAG system around without having to take it yourself - TAGs are harder to count though since I have to notice the beam or get the target info and notice it)
Our side always has 1 ECM since I piloted a trollmando 2D or Craven 3L for the duration of the tests.
13 Games
Us: ECM 2, TAG 0, Them: ECM:3, TAG 1, Outcome: We Lost Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:3, TAG 0, Outcome: We Won Matched Expectation: No
Us: ECM 1, TAG 0, Them: ECM:1, TAG 2, Outcome: We Won Matched Expectation: Inconclusive(1ECM each)
Us: ECM 1, TAG 0, Them: ECM:0, TAG 0, Outcome: We Lost Matched Expectation: No
Us: ECM 2, TAG 0, Them: ECM:0, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:3, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:0, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:2, TAG 1, Outcome: We Lost Matched Expectation: Inconclusive(2ECM each)
Us: ECM 2, TAG 0, Them: ECM:0, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:2, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:3, TAG 0, Outcome: We Lost Matched Expectation: Inconclusive(3ECM each)
Us: ECM 3, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Next 5 games: Done 1 January 2013 All in a craven 3L
Us: ECM 1, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Inconclusive(1ECM each)
Us: ECM 1, TAG 0, Them: ECM:2, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:1, TAG 1, Outcome: We Won Matched Expectation: Inconclusive(1ECM each)
Us: ECM 4, TAG 1, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Yes
Next 11 games: Done 4 January 2013 all games in a craven 3L
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:?, TAG ?, Outcome: We Won Matched Expectation: Inconclusive (Barely saw the other team)
Us: ECM 3, TAG 1, Them: ECM:3, TAG 0, Outcome: We Lost Matched Expectation: No
Us: ECM 2, TAG 0, Them: ECM:2, TAG 0, Outcome: We Lost Matched Expectation: Inconclusive (2ECM each)
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes* (we actually had a friendly disconnect but managed to pull it out anyway)
Us: ECM 3, TAG 0, Them: ECM:0, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:2, TAG 1, Outcome: We Won Matched Expectation: No
Us: ECM 2, TAG 0, Them: ECM:3, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:2, TAG 1, Outcome: We Won Matched Expectation: No
6 January 2012: With DeaconW as wingman
Us: ECM 4, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:3, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
6 January 2012: With Alexei Karnov as wingman\
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Lost Matched Expectation: No
Us: ECM 1, TAG 0, Them: ECM:2, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Inconclusive (2 ECM each)
Us: ECM 2, TAG 0, Them: ECM:2, TAG 2?, Outcome: We Lost Matched Expectation: Inconclusive (2 ECM each)
7 January 2012: With Alexei Karnov as wingman
Us: ECM 1, TAG 0, Them: ECM:0, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:1, TAG 1, Outcome: We Lost Matched Expectation: Inconclusive (2 ECM each)
Us: ECM 1, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Inconclusive (2 ECM each)
Us: ECM 2, TAG 0, Them: ECM:3, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 1, TAG 0, Them: ECM:4, TAG 0, Outcome: We Lost Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:2/3?, TAG 0, Outcome: We Won Matched Expectation: No?
Us: ECM 1, TAG 0, Them: ECM:0, TAG 1, Outcome: We Won Matched Expectation: Yes
Us: ECM 2, TAG 0, Them: ECM:0, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:1, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 4, TAG 0, Them: ECM:2, TAG 0, Outcome: We Lost Matched Expectation: No
Us: ECM 2, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Inconclusive (2 ECM each)
Us: ECM 2, TAG 0, Them: ECM:2, TAG 3, Outcome: We Won Matched Expectation: Inconclusive (2 ECM each)
Us: ECM 4, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:2, TAG 0, Outcome: We Won Matched Expectation: Yes
Us: ECM 3, TAG 0, Them: ECM:4, TAG 2, Outcome: We Lost Matched Expectation: Yes
Tonights games had the added flavour of 2 games with disconnects and one with a missing UI. The only game tonight that I put down as a "No" might actually be inconclusive but I am erring on the side of caution here. The second "No" on the 7th had Garth on our team - regrettably our team was so distracted by having a superstar in the group that we got absolutely steamrolled.
The hypothesis is that the team with more ECM systems will tend to win.
Total Games: 54
Inconclusive Games: 13 - these are games where the number of ECM systems is equal on both teams.
Conclusive games: 41
The conclusive games break down as follows:
Matching expectations: 33
Contradicted expectations: 8
Using the formula given on page 10 of this document:
http://classes.soe.u...nter03/h5m3.pdf
I arrive at
Sigma=0.0619 which is 2.54 matches,
To reach a neutral outcome (20.5/20.5, the results would need to be shifted by 4.92 sigmas),
Assuming the distribution is normal, this gives confidence of > 99.999% - aka "better than 5 nines"
In short, the data still supports the statement that the team with more ECMs on it will tend to win.
Now that we have sufficient mass of data we can start to look at some other interesting features in the set.
The data so far says that you're 4.13x more likely to win if you're on the team that has more ECMs, this seems way too much but again let's assume that it's the case. I now ask the question "how certain is it that you are at least 2x more likely to win the match if you have an extra ECM." I do this by calculating how many sigmas the result can be shifted away from the measured result (33/8) before we arrive at a ratio of 2:1 (here 27.3333/13.6667). The answer is 2.23 sigmas which comes out to 97.4%. This is still a smallish amount of data, but it indicates that
if you have more ECM on your team in a PUG game, you are 97.4% confident that you are at least 2x more likely to win than your opponents.
Deacon and others, please let me know if you spot a problem with what I am doing with the numbers here.
Again, I have asked Garth for some data (~1000 random games) and he wrote back that he's at least going to look into it for me, though the odds of it happening seem pretty slim. Either way I appreciate that our community manager is trying to help me out.
Edit: Addendum 1.
Some people have raised the concern that the results are caused by my bringing an ECM and being a good pilot. In effect they were saying "hey you are raising the average number of ECM on your team and you are a good pilot so you will tend to win more often!" On the surface they make a great point, so let's dig a little deeper:
If you look at the data used (conclusive matches) you'll see that my team had more ECM 2/3 of the time. If you now imagine that I am the best pilot in the universe (really I am only 3rd best, but thanks for the kind thoughts) and would win every match I entered, this would
push the outcome to a 2:1 ratio and not let it go higher The fact that it has come out to >4.1 means that my skill as an individual pilot, even if taken to the extreme it does not explain the outcome. If anything we now need to add that "despite my excellent piloting skill the ECM numbers overcame me"
Still they raised a point that seemed reasonable on the surface so I had to address it.
Edited by Tolkien, 13 January 2013 - 03:20 AM.