Jump to content

Problems With Elo-Hard Stats


88 replies to this topic

#21 scruffy416

    Member

  • PipPipPip
  • 97 posts

Posted 12 March 2013 - 02:28 PM

You all know that you can take a look at a bunch of win/loss data separated out by Chassis in your Profile now, right?

http://mwomercs.com/...stats?type=mech

No recording whether the matches seemed lopsided or not, but I don't think you lose too much if you separate that out of the equation.

#22 freak

    Member

  • Pip
  • 14 posts

Posted 12 March 2013 - 02:31 PM

Saw that, thanks scruffy416.

Was more the ratio of casualties I wanted to draw attention to, I don't mind the losing (god knows I do it enough) it's the Landslides that depressing and looks bad for new players too.

#23 FerretGR

    Member

  • PipPipPipPipPipPipPipPip
  • Legendary Founder
  • Legendary Founder
  • 1,445 posts

Posted 12 March 2013 - 02:32 PM

View PostWolvesX, on 12 March 2013 - 01:54 PM, said:

ELO is not working for a team game.

http://en.wikipedia....gs_beyond_chess

(football, for example, is a team game)

OP: Two things. You don't need to present your credentials before you make an argument. Even when arguing a perfectly reasonable perspective, doing so is including an argument from authority, and for me at least, it puts a black mark on the argument. Your argument, if it's strong, should stand alone.

Second, I don't know if you read any of the fluff, but there's a concept called "combat loss grouping" that talks about how when one mech is lost, chances are all of the other mechs in combat at or around that mech's weight class are close to being lost, and their loss is inevitable if the combat isn't ended. I think there's a similar mechanism at work in MWO, and that's what's responsible for your "unbalanced" scoring games. In MWO, though, I think the way it works is that when mechs start to drop on a team, it tends to start a domino effect and the rest of the mechs will drop faster and faster. An 8-0 or 8-1 game is a steamroll, I'll give you that, and sometimes that's just the way it goes. I think when you start to get into the 8-2, 8-3 range, though, you'll see, examining how the games went, that these games were actually pretty close, but when one mech dropped a bunch dropped in succession as the enemy team focused down on fewer and fewer mechs. It's similar to CLG in that if a team is getting beaten up a bit, the damage is probably being distributed fairly evenly, and as mechs drop, that same level of damage is being spread over a smaller and smaller number of increasingly beat up mechs.

What starts the cascade? Who knows? One small mistake, IMHO, and not a whole team of morons... someone rounds a corner and gets a faceful of Splatcat, a scout stumbles into the team, a lucky headshot, what have you. But an advantage in this game leads to a bigger advantage for players who capitalize.

Edited by FerretGR, 12 March 2013 - 02:47 PM.


#24 freak

    Member

  • Pip
  • 14 posts

Posted 12 March 2013 - 02:37 PM

Yes but Footballers can see/hear/communicate with each other without a keyboard (even if only to grunt -joke)

they also get paid a hell of a lot more then me (sigh) ;)

Oh and for those who might be curious, according to my stats I've have 80 matches in Assaults, 4 in Mediums and 1 in a light since the patch so yeah, if it doesn't have me pegged as an Assault pilot by now...... :)

#25 Royalewithcheese

    Member

  • PipPipPipPipPipPipPipPipPip
  • 2,342 posts

Posted 12 March 2013 - 02:40 PM

View Postfreak, on 12 March 2013 - 02:31 PM, said:

Saw that, thanks scruffy416.

Was more the ratio of casualties I wanted to draw attention to, I don't mind the losing (god knows I do it enough) it's the Landslides that depressing and looks bad for new players too.


IMO it tends to snowball sometimes. You get chains of events like: 3-L pilot thinks the 3-L is invincible and rushes enemy -> Atlas and Hunchback wander off after the 3-L's blue triangle -> 3-L turns off not to be invincible -> Atlas and Hunchback meet entire enemy team -> match is now 5v8.

#26 Jestun

    Member

  • PipPipPipPipPipPipPipPip
  • 1,270 posts

Posted 12 March 2013 - 02:41 PM

View Postscruffy416, on 12 March 2013 - 02:28 PM, said:

You all know that you can take a look at a bunch of win/loss data separated out by Chassis in your Profile now, right?

http://mwomercs.com/...stats?type=mech

No recording whether the matches seemed lopsided or not, but I don't think you lose too much if you separate that out of the equation.


"No BattleMech statistics available."

lol...

#27 Tangelis

    Member

  • PipPipPipPipPipPip
  • Elite Founder
  • Elite Founder
  • 442 posts
  • LocationMontreal

Posted 12 March 2013 - 02:41 PM

Must admit you put in a lot of time and hard numbers to argue your point. Numbers do not lie and thus you have factual hard data. Good post in my opinion.

The only nitpick I have that really jumped out at me was the following:

View Postfreak, on 12 March 2013 - 01:52 PM, said:


The Win/Lose results were as follows,

Wins = 6
Loses = 14

That's a lose rate of 70%, if the ELO system is supposed to provide me with balanced matches, why isn't closer to 50%???



Two players can be "equal skill" level but there are several game mechanics that create variables that any ELO system can not factor. For example: Critical hits..ammo explosions..lucky head shot.. and the like. You and I can drop in a 1 on 1 match with an identical ELO rating, after ten games however we will likely not be 50/50 on the wins losses. Having a loss rate of 70% does not mean that the ELO system is not working.

Also remember that the ELO system went live to test on a mass level. It stands to reason adjustments need to be made as time goes on to balance it out accordingly. It was even mentioned either in the patch notes or a Dev update post or something to that effect that it was expected changes would need to be made.

ELO (If I remember reading it right, If I am wrong please correct me) but ELO is not only based on the player but the chassis you drop in. If those 20 games were the first 20 you dropped in using that awesome (Or any other chassis) it's creating a whole new number. So a mech you have used 100+ times is going to generate a more accurate ELO as opposed to one you have just started using. This is of course since ELO was implemented only, not your entire career in MWO.

I agree the system needs tweeks and such but I don't think it's as all out bad as many believe. Considering the differences in playtimes of hard core players vs casual and the fact the system has only been live a short period of time... the casuals are likely to get thrown all over the board until thier ELO plateau's. A Lucky winning or hard luck losing streak can make a big difference.

#28 Matthew Craig

    Technical Director

  • 867 posts
  • LocationVancouver, BC

Posted 12 March 2013 - 02:42 PM

freak: I'd also point out that it's really hard to determine from the number of Mechs alive how balanced the game was, I've been in games that ended 8/1 that were really close up until the point where one side started to gain an advantage.

It's a natural consequence of Mechwarrior that once one team starts to gain an advantage the tonnage disparity can quickly result in one side struggling to get back into the fight. Just balancing straight up on tonnage as you suggest though would not balance the game as pilot skill does play a significant part.

If you take two teams of identical Mechs so zero tonnage disparity the outcome is likely very one sided if one team has a vast difference in skill over the other. In the end creating balanced matches is a complex problem, we have every confidence that the new system will create great matches given time to tune/balance it.

#29 ObsidianSpectre

    Member

  • PipPipPipPipPipPip
  • 289 posts

Posted 12 March 2013 - 02:46 PM

Thanks for the comments on this Craig. The Elo system doesn't feel perfect yet, but it does feel much better than what we had before and honestly I'd rather find quicker matches than perfect ones, so imperfection in exchange for faster matches is something I'm fine with.

#30 SgtKinCaiD

    Member

  • PipPipPipPipPipPipPipPip
  • Overlord
  • Overlord
  • 1,096 posts
  • LocationBordeaux

Posted 12 March 2013 - 02:47 PM

Finally some answers, thanks !

#31 FerretGR

    Member

  • PipPipPipPipPipPipPipPip
  • Legendary Founder
  • Legendary Founder
  • 1,445 posts

Posted 12 March 2013 - 02:49 PM

View PostMatthew Craig, on 12 March 2013 - 02:42 PM, said:

freak: I'd also point out that it's really hard to determine from the number of Mechs alive how balanced the game was, I've been in games that ended 8/1 that were really close up until the point where one side started to gain an advantage.

It's a natural consequence of Mechwarrior that once one team starts to gain an advantage the tonnage disparity can quickly result in one side struggling to get back into the fight. Just balancing straight up on tonnage as you suggest though would not balance the game as pilot skill does play a significant part.


Yes, this is exactly what I was trying to say in my post, if it wasn't clear.

#32 DragonsFire

    Member

  • PipPipPipPipPipPipPip
  • Wrath
  • Wrath
  • 655 posts

Posted 12 March 2013 - 02:50 PM

View PostMatthew Craig, on 12 March 2013 - 02:42 PM, said:

freak: I'd also point out that it's really hard to determine from the number of Mechs alive how balanced the game was, I've been in games that ended 8/1 that were really close up until the point where one side started to gain an advantage.

It's a natural consequence of Mechwarrior that once one team starts to gain an advantage the tonnage disparity can quickly result in one side struggling to get back into the fight. Just balancing straight up on tonnage as you suggest though would not balance the game as pilot skill does play a significant part.

If you take two teams of identical Mechs so zero tonnage disparity the outcome is likely very one sided if one team has a vast difference in skill over the other. In the end creating balanced matches is a complex problem, we have every confidence that the new system will create great matches given time to tune/balance it.


I have been in rounds where it was just that, a seeming 8-1 stomp, but it ended up being very close as many on the other team were only half functional or 1 shot away from dead. I've also been on the winning side of the 8-1 in that scenario as well. It can really come down to a few key moments in any match to wildly swing the weights of balance to the other team. It could be something as simple as me at 100% armor noticing one of my assaults getting hammered and stepping in front to draw fire to allow him to retreat, reset, and then recommit to the fight.

As Matthew noted in an earlier post as well, they haven't even done a first tuning pass on it yet and I have already seen far more balanced and overall engaging rounds than I had prior to Elo. This is speaking as someone who plays lone wolf as well as 2, 3, and 4 man drops. I realize that not everyone is getting that experience, but give it a bit of time folks, it's getting there.

Edited by DragonsFire, 12 March 2013 - 02:52 PM.


#33 MechGorilla

    Member

  • PipPip
  • Legendary Founder
  • Legendary Founder
  • 33 posts
  • LocationFishers, Indiana, USA

Posted 12 March 2013 - 02:54 PM

View PostJestun, on 12 March 2013 - 02:06 PM, said:

If I tossed a coin 20 times it's unlikely to actually be 10 x heads and 10 x tails. And that only has 2 potential outcomes!


A coin flip also doesn't have a software system expressly designed to balance out the odds of the coin landing on one side or the other. Which is what the OP's post was about. I'm not saying his conclusions are correct, but this is not a good argument against his conclusions.

#34 Lyrik

    Member

  • PipPipPipPipPipPipPip
  • FP Veteran - Beta 1
  • 568 posts

Posted 12 March 2013 - 02:55 PM

a ) 24 years of BT are meaningless in MWO
b ) 20 games are not enough data.
c ) Perhaps you just suck in your current mech. If you lose enough, your ELo will drop and you will be matched with players of your skilllevel.
d ) Elo is not a perfect system!!!!!!!!
e ) The devs stated that it will be adjusted.
f ) 20 games are not enough data.
g ) Battletech experience is nearly worthless in this game ;-)

#35 ciller

    Member

  • PipPipPipPipPipPip
  • Liquid Metal
  • Liquid Metal
  • 486 posts
  • LocationEdmonton

Posted 12 March 2013 - 02:55 PM

If its worth anything, Ive had hundreds upon hundreds of matches in my medium and when I am in a 4-man group I find that we can still pull a consistently high win average (+90%) compared to when I am just pugging it (around 50%) since ELO was patched in.

Perhaps its because we automatically have 4 skilled people on one team or perhaps its because we have 4 people working together over voice-coms but either way, ELO doesn't really work when I am in a group - almost all my games seem overly easy.

#36 FerretGR

    Member

  • PipPipPipPipPipPipPipPip
  • Legendary Founder
  • Legendary Founder
  • 1,445 posts

Posted 12 March 2013 - 02:58 PM

View PostMechGorilla, on 12 March 2013 - 02:54 PM, said:

A coin flip also doesn't have a software system expressly designed to balance out the odds of the coin landing on one side or the other. Which is what the OP's post was about. I'm not saying his conclusions are correct, but this is not a good argument against his conclusions.


Sure it is. The point is that the sample size of the OP is FAR too small to be indicative of anything. He should continue to collect data for a month or two and see how that works out.

As an aside, a coin is pretty close to perfectly balanced in almost every case... much closer to guaranteeing 50/50 than any software could ever do.

#37 Theevenger

    Member

  • PipPipPipPipPip
  • 194 posts
  • Facebook: Link
  • LocationCalifornia

Posted 12 March 2013 - 02:59 PM

Your data is all well and good, but I have only had 2 matches that were 8/0 since they implemented it. Before they implemented it, I was often get 5 matches in a row that were 8/0. ELO is working great and I have found almost every match to be challenging since it was implemented. I could not be happier with how it is working out, except maybe if I had an ELO rating for each chassis type (or even individual Mech) rather than one per weight class.

#38 Jestun

    Member

  • PipPipPipPipPipPipPipPip
  • 1,270 posts

Posted 12 March 2013 - 02:59 PM

View PostMechGorilla, on 12 March 2013 - 02:54 PM, said:


A coin flip also doesn't have a software system expressly designed to balance out the odds of the coin landing on one side or the other. Which is what the OP's post was about. I'm not saying his conclusions are correct, but this is not a good argument against his conclusions.

I'm not arguing against his conclusions, I'm arguing the validity of the data sample which he used to come to those conclusions.

#39 Nidhoggr

    Member

  • PipPip
  • Overlord
  • Overlord
  • 44 posts
  • LocationSTL, USA

Posted 12 March 2013 - 03:01 PM

The Elo system has been working perfectly in my experience. And just to hammer this point in further, 20 matches is not an adequate sample size for this sort of analysis. You have to remember that the total population out of which you are drawing your sample from is in the millions (total matches played over all players). Everyone who suggests on this forum that it would be better to have the matches randomly determined are essentially saying that they would prefer to go through the world blind rather than have sight. Elo is equivalent to giving the matchmaker process sight. The sight might not be perfect at the moment, but with more measurements over time, the vision will be 20/20 in a few months.

#40 zmeul

    Member

  • PipPipPipPipPipPip
  • 397 posts
  • LocationBuzau, Romania

Posted 12 March 2013 - 03:07 PM

I had my own mischief with the ELO rating until it settled me to the correct bracket: http://mwomercs.com/...86#entry1983686

Posted Image

but, I see 2 major issues that affect one's ELO rating and matchmaking quality
  • rating based on class; should be per chassis, or even per variant
  • solo vs grouping
I play exclusively my D-DC, if I decide to play something else assault class, my rating and matchmaking quality will go to new places ;)

Edited by zmeul, 12 March 2013 - 03:10 PM.






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users