202 replies to this topic

[MischiefSC]

Bhael Fire, on 08 April 2014 - 07:53 AM, said:

The real issue is that an average solo player (like myself) can get caught in an Elo funnel where they get artificially pulled up or down into an Elo bracket they don't belong in because of a string of lucky (or unlucky) matches...and get stuck there for an inordinate amount of time because of the actions of the 11 other random players. I was pointing out the fallacy that a player's Elo score will eventually settle into its rightful place, when in fact it can (and does) fluctuate wildly on a regular basis ...especially if you play solo a lot.

For example my current w/l ratio is about 1.3...and it's rising. I can assure you, this has nothing to do with my skill as a player. And as such, my matches lately have been brutal...I'm completely out of my element.

I'm going to try and avoid coming in heavy to this thread because I've done it dozens of times and at a certain point it becomes more like arguing religion than reality. Elo (or more to the point the Elo-based system MW:O uses. It's not actually Elo) works and works just fine in this sort of environment and that's been proven, shown, demonstrated, confirmed and is otherwise not even something that is reasonable to debate any more than 1+1 = 2 is reasonable to debate.

Now, you can debate how many matches are required to seat you. You can debate how much your personal swing in performance can 'unseat' your position as you learn - you can debate how much your Elo should vary based on chassis and loadouts. It's theoretically quite possible to create a more complex, granular modifier for Elo based on the chassis, loadout and even team compositions of yours vs the other teams to get you an even more precise seeding.

With the right telemetry and a sufficiently complete model of your history with chassis, loadout and team composition it would be possible to rank every single player in MW:O exactly and precisely in a 1-30,000 player ladder. You'd know who the 'best' was, second, etc.

We don't need that though. Due to population density and necessity of some form of weight/chassis matching you're doing good to have a decent match in chassis and to fill 24 players in a given skill band within 2 minutes.

I realize that it feels like your win/loss is swinging around. It does - unless you play the same chassis and loadout all the time in the same weight-class you're going to get some variance.

That's why your Elo score moves relatively slowly. It'd take dozens of atypical wins or losses to make a significant move in your ranking. You'll have big swings sometimes. Flip a coin ~1024 times and in that process you'll get 'heads' 10 times in a row - yet at the end of the 1024 flips you'll have ~512 heads and ~512 tails.

It's alright. Short-term variance does not equate to long-term statistical instability.

[Deathlike]

aniviron, on 07 April 2014 - 09:38 PM, said:

I don't have a sauce, but I seem to remember a PGI employee saying that each server runs just one map; it could be that it tries to balance server load, and fills capacity on one before shifting to another. Matches take about the same amount of time, and so you get stuck in a cycle of being on the same server/map over and over. But that could be me misremembering, too.

I hope you find that source soon™.

It can explain a few things.

For instance, once you "end the match" on the virtual server, it is "easy" and "also lazy" to create the same server again with the same map (if not the same mode to boot), assuming you've gotten the proper ratios correctly applied across map distribution and mode distributions. Of course, it assumes the launch button "reacts quickly enough" to add you back onto said server. It wouldn't be too far-fetched and the distribution reality might not be too far off either.

It would to some degree "reduce" the actual randomization of stuff happening (like the map weighing which is being done to separate the Night/Snow maps from the original versions to "even them out"). So, getting stuck on the same map can actually be MORE PREDICTABLE...

I wouldn't say that's a good thing either.

[Flaming oblivion]

I think this problem would be solved by lowering it from 12 v12 to at most 6v6 maybe 1-2 v1-2 is where it really needs to be, On another note believe me the " noobs" enjoy being steam rolled as little as you do, also pre made drops are part of the problem with PuG's being steam rolled. 25% of you team being "noobs" in a 12v12 is almost a guaranteed loss. 25% in a 4v4 isn't such a big deal. Just my thoughts

[Amsro]

Prosperity Park, on 06 April 2014 - 08:53 AM, said:

Please resolve this for me:

If your Elo Rank it too high, then you get mixed with crappy players who can't fight worth crap and shoot LRMs into walls at point-blank,

AND

If your Elo Rank is too high, then you only play with Meta Elitists and the games become overly competitive and one-dimensional with precision sniping being the norm.


So, which is it?

It's both, thats the point. 1400 Elo variance is the max spread meaning you can literally be a soother sucking hello I just started player and be in the same match as a 10,000 match veteran.

This is the main problem with matchmaker, is that it is trolling EVERY SINGLE PERSON in the game, at the same time no less. ! Amazing isn't it.

[Spr1ggan]

Khobai, on 05 April 2014 - 12:36 PM, said:

this is why we need 4v4 and 8v8 gamemodes. so individual skill matters more than the RNG of who gets put on your team.

also the 12v12 gamemodes need to split teams up into lance vs lance battles to make individual skill matter more. Its too hard to carry in 12v12 because 1 person cant beat a deathball of 6+ enemy mechs on their own.

I agree and most of the maps on the game aren't fit for 12v12 anyway. Forest Colony and Forzen City should be 4v4. The bigger ones 8v8 and 12v12. Although i still don't think there's a map currently in game that suits 24 players.

[Bhael Fire]

MischiefSC, on 09 April 2014 - 02:36 PM, said:

It's alright. Short-term variance does not equate to long-term statistical instability.

Actually, it's not alright. I've been playing regularly nearly every day since before Elo was put into the game, so I'm pretty sure I've been here long enough for my Elo score to have settled in its proper place.

Going weeks at a time with consistently bad matches is not a "short term variance" — it's a major flaw in matchmaking that leads to a frustrating game experience. Luckily, I'm addicted to this game despite all of its shortcomings.


So, you are incorrect about Elo working effectively; it is not "working just fine" nor has it been "proven, shown, demonstrated, confirmed" to be working just fine — at least not within the parameters of what I and many other players consider to be "just fine."

I'm sure, mathematically speaking, it might look like it's just fine...but in reality...not so much. So, it's not a case about 1+1 as much as 1+Pudding.

That's one of the reasons they plan on re-working it in the future; it's not working for a large portion of the population.

[Abivard]

Such small minds.

Just because something is wet doesn't make it water.

Popular myth has it the first wheel was chipped from stone, that does not mean a modern hi tech tire and the stone tire are the same thing. You are not going do very well with stone tires on your race car are you? Of course not.

Your car with stone wheels may eventually get you to the finish line but what good will it have done you or anyone else.

1+1=2 that can said to be Elo as invented, what we have is 12+12=24, if you keep saying the answer is still 2 you're a fool.

Mischief fails to see this, he always has. He makes long winded statements saying everything is settled, no discussion should be allowed, he has seen the real facts and you will just have to trust him on it. MischiefSC does not want you seeing or reading or knowing anything contrary to his point of view.

Not a single reputable person has ever made the claim that Elo created the perfect system, but we have Roadkill and MischeifSC insisting Elo did and EVERYONE knows this, end of discussion.

We really haven't a clue what 'system' PGI uses beyond that it is an Elo like system, which pretty much can mean anything whatsoever. PGI has always been rather free and loose with their ideas. But the one thing both I and PGI seem to be in agreement with is that Elo systems as it was would not work in MWO, but what PGI put in is not seeming to be working either.

Now just maybe if everyone was stuck with the exact same mech, with the exact same build, on the exact same map, with the exact same mode, after a few hundred or thousand or so games, it might be able to properly place you.

To make the claim that I and everyone in the world plays at the same skill level no matter what mech or build or mode we play is ludicrous.

The fact: that PGI assigns an Elo rating for different weight classes should give you a hint that something is fishy.

The fact: that no one in any of the chess Federations or leagues uses Elo's original formula. It was however a great starting point for 1 on 1 games, but that is all it was, a starting place.

The Fact: that so many people, institutions and business' are working on better systems to rate players in Massive multi-player games. (remember, multiplayer can mean anywhere from 2 and up players. ie. Chess is a multiplayer game, keep it in context)

Glicko was based off of the Elo theory, but it is not Elo.

Trueskill was based off of Glicko's theorems but is neither Glicko or Elo.

Pearls before swine in Roadkill and MischeifSC case I am afraid, but let not their Vocal and strident ignorance cloud your mind, go read and research for yourself.

Edited by Abivard, 09 April 2014 - 03:37 PM.

[Mystere]

What is this 1vs1, 2vs2, and 4vs4 nonsense being spouted? I and many others want battalion-sized or larger formations. But, we are still willing to wait until Community Warfare comes out, hopefully in the near future.

Hell, I think it's really time to ask, do people want MWO to just be an arcade-style, arena-style, Solaris VII themed eSport? Or do people want it to be a bloody, brutal, Mechwarrior game in which armies of big stompy walking machines of death face each other? Because I did not go here for the former.

[Flaming oblivion]

^ Your entitled to your opinion and your wish no one is saying that 12v12 should be removed, simply that their should be other options to cater to all , I tell you without doubt in my mind 12v12 being the only option is not helping them hold onto new players. It's simply to "brutal" to use your term

[Mystere]

Flaming oblivion, on 09 April 2014 - 05:01 PM, said:

^ Your entitled to your opinion and your wish no one is saying that 12v12 should be removed, simply that their should be other options to cater to all , I tell you without doubt in my mind 12v12 being the only option is not helping them hold onto new players. It's simply to "brutal" to use your term

Who said anything about removing 12vs12?

Also, Could it be possible that people actually want something like 36+vs36+ or (gasp!) an asymmetrical XvsY?

1vs1, 2vs2, 4vs4, they all sound like eSport to me. I did not go here for that.

If people really want an eSport, then let's have a 24(+)-player last-man-standing no-holds-barred winner-takes-all game mode, especially because:

[MischiefSC]

Bhael Fire, on 09 April 2014 - 03:19 PM, said:

Actually, it's not alright. I've been playing regularly nearly every day since before Elo was put into the game, so I'm pretty sure I've been here long enough for my Elo score to have settled in its proper place.

Going weeks at a time with consistently bad matches is not a "short term variance" — it's a major flaw in matchmaking that leads to a frustrating game experience. Luckily, I'm addicted to this game despite all of its shortcomings.


So, you are incorrect about Elo working effectively; it is not "working just fine" nor has it been "proven, shown, demonstrated, confirmed" to be working just fine — at least not within the parameters of what I and many other players consider to be "just fine."

I'm sure, mathematically speaking, it might look like it's just fine...but in reality...not so much. So, it's not a case about 1+1 as much as 1+Pudding.

That's one of the reasons they plan on re-working it in the future; it's not working for a large portion of the population.

Is your win/loss over 1.0? I'm going to bet probably. If that's the case then on average you're actually winning more than you're losing.

Even if MW:O used a refined TrueSkill system and your score was 100% accurate to the mech, build, loadout and team compositions you'd still have matches pretty much exactly like you do now. You'd have teams where teammates make poor choices, you'd still have 12-1 stomps. It wouldn't change much of anything - not until you've got millions of players and a hundred thousand concurrent players to choose from to ensure a far more balanced match.

In which case you'd still have people playing a bad game vs a good game and end up with one-sided stomps.

I'm curious; what are you looking for? Are you looking for the system to identify when you've had a run of bad matches and intentionally throw you into a match where you have a significant score advantage? What are you looking for the matchmaker to do?

Because currently all it does is put you in a team of 11 other players in a match against 12 other players where it's matched, as accurately as the current population allows from among only those people who hit 'launch' within 2 minutes of when you did and min approximately weight-matched teams.

Is that not what you want? What, sincerely, are you looking for?

Abivard, on 09 April 2014 - 03:33 PM, said:

Such small minds.

Just because something is wet doesn't make it water.

Popular myth has it the first wheel was chipped from stone, that does not mean a modern hi tech tire and the stone tire are the same thing. You are not going do very well with stone tires on your race car are you? Of course not.

Your car with stone wheels may eventually get you to the finish line but what good will it have done you or anyone else.

1+1=2 that can said to be Elo as invented, what we have is 12+12=24, if you keep saying the answer is still 2 you're a fool.

Mischief fails to see this, he always has. He makes long winded statements saying everything is settled, no discussion should be allowed, he has seen the real facts and you will just have to trust him on it. MischiefSC does not want you seeing or reading or knowing anything contrary to his point of view.

Not a single reputable person has ever made the claim that Elo created the perfect system, but we have Roadkill and MischeifSC insisting Elo did and EVERYONE knows this, end of discussion.

We really haven't a clue what 'system' PGI uses beyond that it is an Elo like system, which pretty much can mean anything whatsoever. PGI has always been rather free and loose with their ideas. But the one thing both I and PGI seem to be in agreement with is that Elo systems as it was would not work in MWO, but what PGI put in is not seeming to be working either.

Now just maybe if everyone was stuck with the exact same mech, with the exact same build, on the exact same map, with the exact same mode, after a few hundred or thousand or so games, it might be able to properly place you.

To make the claim that I and everyone in the world plays at the same skill level no matter what mech or build or mode we play is ludicrous.

The fact: that PGI assigns an Elo rating for different weight classes should give you a hint that something is fishy.

The fact: that no one in any of the chess Federations or leagues uses Elo's original formula. It was however a great starting point for 1 on 1 games, but that is all it was, a starting place.

The Fact: that so many people, institutions and business' are working on better systems to rate players in Massive multi-player games. (remember, multiplayer can mean anywhere from 2 and up players. ie. Chess is a multiplayer game, keep it in context)

Glicko was based off of the Elo theory, but it is not Elo.

Trueskill was based off of Glicko's theorems but is neither Glicko or Elo.

Pearls before swine in Roadkill and MischeifSC case I am afraid, but let not their Vocal and strident ignorance cloud your mind, go read and research for yourself.

You know I was going to start with something as equally snarky and disingenuous as your whole post; linking you to some online classes for analytics and statistics.

In the end though that's pretty pointless and I get that.

Math is hard. I'm not being sarcastic there; it is hard. I should know, it being hard is why I get paid well.

What you're arguing is that the fundamental basis of analytics and statistics is false. It really is what you're trying to say here. I get that you're trying to dodge around that and otherwise use a bunch of postmodernist justifications why the fundamental basis of statistical analysis and analytics just falls apart in 12 v 12 matches in MW:O. This is quite literally akin to saying that subtraction doesn't work if you're subtracting numbers that are written in different languages. Yes, yes it does work. The language the numbers are written in is irrelevant.

I've linked you before, repeatedly, to the formulas behind Glicko and TrueSkill. Both use Elo as their foundation. Their purpose is, in the case of Glicko, to provide more accuracy in environments with more variables and in TrueSkills case to take your performance in one game and apply it to another. TrueSkill also uses Elo as a foundation to actually place you in a stack ranking instead of as an estimated score. Both are excellent systems. Both, like the MM in MW:O, use the Elo equation as a basis from which to draw their data.

All Elo is in this regard is an equation whos purpose is to derive the probability of you winning a giving event based on how often you've won a given event in the past, all based on the exact same probability result of the other player (s). That's it.

You've had all this explained repeatedly. I get that you seem to have a vested interest in not wanting to understand or that somehow it just... doesn't work. It does work, the reason why it works is the exact same basis as why all statistical analysis works. It's identifying the impact of a single consistent variable across multiple samples all solving for the same result (win/loss) in an environment with a set number of players.

The MW:O MM and its use of Elo is NOT exact - it's only moderately accurate. Your performance will vary by chassis and loadout; in fact it'll vary as well by the relative loadout of your team and the other team. As an average though it's a solid indicator for about where you sit on average.

Which is exactly what the MatchMaker needs in MW:O. It's not trying to put you in a stack ranking - you're not the 114th best player in MW:O. It's not even trying to fit you exactly; it's not looking to match your 1624 Elo with 23 other people with a 1624 Elo, give or take a couple of points.

All it's trying to do is take your approximate skill (crudely adjusted for weight class of mech) and put you in matches with and against people of comparable skill. My biggest complaint, matching high/low to a goal has been addressed by having 3 tiers. You're going to be matched to a band. My other big concern, the skewing of results based on that of playing in premades, are resolved by A) matching to an Elo band where a disparity of your actual Elo vs your estimated Elo could be 200 points off doesn't matter anymore really and B ) the tiny percentage of premade drops vs pug drops.

I'm not going to change your mind Abivard. Doesn't matter if I do or don't. I'm not going to spend hours explaining yet again why you're wrong and how and the specific data behind it. I've done so repeatedly and it's not a useful expenditure of time; you're invested in it being wrong. Okay. Well, best of luck to you. If you do manage to actually prove that the Elo equation is incapable of solving for a single variables performance over hundreds of samples in a 12 v 12 environment (or any environment for that matter) I hope you publish it. You'd literally be giving not just PGI a black eye but Microsoft, Google, Facebook, Twitter, EDS, Yahoo.... it's a long list of people who use analytics like the Elo equation to distill out of complex interactions the impact a single consistent or even semi-consistent variable.

We know that's not going to happen though because your argument that it doesn't work is all anecdotal and comes down to 'it just doesn't feel like Cricket'. Okay. Well, great. I'm sorry it just doesn't feel right to you. The math is correct though. I'm not going to give you 2 years worth of statistics and analystics education on this forum, so either trust it or don't, or go do some actual legitimate research.

How about this - why don't you e-mail Jeff Moser about the viability of an Elo-based ranking system in a 12 v 12 pug based game like MW:O. The guy knows his stuff and is very familiar with TrueSkill - he helped code it if I remember correctly. I get that you don't know me and I'm not going to blow personal privacy by outting myself on a game forum but he's a known web presence and is about as close to a known expert on the topic as you will ever find.

e-mail him. Go ahead - say you're concerned that people use an Elo-based matchmaker for multiplayer games to identify a single players performance out of a pug team and you don't think that can actually work. He's a busy guy and will probably just shoot you some links but that's a nice 3rd party who is undeniably an expert on the topic you can touch base with.

Or you can stay here and argue a point that, if you understood statistical analysis and analytics, would just look silly.

Best of luck to you. I'm not even playing MW:O at the moment; the ARM update for Kerbal Space Program is out and I'm building a rock garden at KSC using captured asteroids.

[Roadkill]

Abivard, on 09 April 2014 - 03:33 PM, said:

read and research for yourself.

It's laughable that you would suggest this, Abivard, because the only possible conclusion that someone would come to if they read all of the research that's been posted is that you are wrong.

[Roadkill]

Bhael Fire, on 09 April 2014 - 03:19 PM, said:

So, you are incorrect about Elo working effectively; it is not "working just fine" nor has it been "proven, shown, demonstrated, confirmed" to be working just fine — at least not within the parameters of what I and many other players consider to be "just fine."

Bhael Fire, your complaint is valid but you're leveling it at the wrong system. You even responded to the post that should have caught your attention.

No matter how accurate your skill ranking is; no matter how perfect the system is that's generating those skill rankings; no matter how fast your skill ranking converges on your true skill... none of that matters if the matchmaker basically throws it all out the window and selects players from a 1400-point ratings range.

That's not a problem with Elo. That's a problem with the matchmaker, which is what I've been saying all along.

[Bhael Fire]

Roadkill, on 09 April 2014 - 07:47 PM, said:

That's not a problem with Elo. That's a problem with the matchmaker, which is what I've been saying all along.

That's what I've been saying too; Elo is wrong for a game like MWO.

MischiefSC, on 09 April 2014 - 06:52 PM, said:

If that's the case then on average you're actually winning more than you're losing.

Ah man, I could have saved you a LOT of wasted time typing if I could have stopped you after this sentence. I think I see the source of your confusion about how a matchmaker should work in a game like MWO:

Winning ≠ Good Match

And yes, of course my W/L ratio is high; that's the problem. It's inordinately high (around 1.3 for weeks now and getting higher) because I keep getting matched with players above my pay grade. The game thinks it needs to "challenge" me, when all it's doing is tossing me in meat grinder, after meat grinder; I often die regardless if my team wins or not.

That's the issue with Elo in a game like MWO. As I explained earlier, it's too easy to fall into Elo "funnels" (as I call them) because the game equates "winning" with player skill...which is NOT even remotely in the same ballpark when we're talking about a game that puts 11 random people on your "team."

So, it's entirely possible to get caught on a higher Elo "plateau" where, because of your Elo, you are forced to fight other players that are better than you or at the very least just as good. So...while on this plateau you get batted around match after match as the really talented players and groups dance the 50/50 dance....until you eventually get caught in downward funnel to a plateau closer to your personal skill level.

So to answer your question, what do I want from the matchmaker? Well it has nothing to do with "winning" because winning is meaningless in this game. It means nothing at all (at least not in it's current form).

What do I consider to be a good match? There are a lot of things that I think contribute to a good match, the least of which is dying match, after match...after match without any kills....for days....weeks on end.

That's not "fun" and even if you "win" it still makes for a frustrating gaming experience when it happens as frequently as it does in MWO.

[meteorol]

MischiefSC, on 09 April 2014 - 02:36 PM, said:

...it would be possible to rank every single player in MW:O exactly and precisely in a 1-30,000 player ladder...

You accidently hit the "0" key one time too often and misplaced the comma

[Thorqemada]

50/50 is a sign that ELO works in the Big Picture bcs fighting equal opponents will end up close to a 50/50 ratio if the amount of samples is big enough.

Though the way to achive this 50/50 ratio is a pretty rough experience and leaves room for further improvements and changes.

[Roadkill]

Bhael Fire, on 09 April 2014 - 08:12 PM, said:

That's what I've been saying too; Elo is wrong for a game like MWO.

Aha, that's an important distinction. If I understand correctly, you aren't really saying that Elo doesn't work here. You're saying that it doesn't really matter because it's not right to try and use it in a game like MWO in the first place.

Is that a fair summary? That I can understand.

That's really just another way of saying that the fundamental problem is the matchmaker, not Elo. The matchmaker isn't giving us good matches because it's using the wrong tools to put them together.

[Bhael Fire]

Roadkill, on 10 April 2014 - 07:54 AM, said:

Is that a fair summary? That I can understand.

That's really just another way of saying that the fundamental problem is the matchmaker, not Elo. The matchmaker isn't giving us good matches because it's using the wrong tools to put them together.

Yes. Exactly. Elo is not appropriate in this game because it makes several erroneous assumptions from the get-go....the most egregious of which being that "winning" equates to individual skill and/or a fun gaming experience.

Truthfully, I think it would just better if the game matched players on a 1:1 basis based on mech tonnage (individually, not team tonnage) and used K/DR tiers instead of Elo tiers....since those are the two most important factors in this game that are indicative of overall threat level and an individual player's skill. Perhaps average damage done could factored into that equation, as well. The point is, "winning" is the least important factor in this game when it comes to appraising player skill...At least in the current game anyway.

And as I mentioned, that's why PGI plans on taking these things into account eventually when revising matchmaker. It's just painful waiting in the interim.

[Flaming oblivion]

You cant be saying the matchmaking system pits people against each other based off of wins, As I've said in other threads i'm new but even I know this would be a bad idea just looking at the score sheet at the end of the match , Your not telling me the 3 guys at the end of every match that have done 400-1k damage are on the same skill level as the 3 guys that did 0-150 (on the winning side). wins cant be the decider of who's fit to face who. It may work if battles were smaller because you'd have to do your bit to compete but not in this madness that is 12v12.

Edited by Flaming oblivion, 10 April 2014 - 09:57 AM.

[Joseph Mallan]

Flaming oblivion, on 10 April 2014 - 09:56 AM, said:

You cant be saying the matchmaking system pits people against each other based off of wins, As I've said in other threads i'm new but even I know this would be a bad idea just looking at the score sheet at the end of the match , Your not telling me the 3 guys at the end of every match that have done 400-1k damage are on the same skill level as the 3 guys that did 0-150 (on the winning side). wins cant be the decider of who's fit to face who.

I have been both of those players on the winning and losing side Oblivion... you can have great games and terribad games.