65 replies to this topic

[Fabe]

Another problem with the Elo is that if I recall correctly it only looks at whether or not you were on the winning team which isn't a real good indication of individual skill. Put a poor player like me on a team good enough to carry me and even if I don't actually improve my Elo score will still go up just for being on the winning team.

[DeadlyNerd]

Fabe, on 16 August 2013 - 05:33 AM, said:

Another problem with the Elo is that if I recall correctly it only looks at whether or not you were on the winning team which isn't a real good indication of individual skill. Put a poor player like me on a team good enough to carry me and even if I don't actually improve my Elo score will still go up just for being on the winning team.

Liked for honesty.

[Hauser]

DeadlyNerd, on 16 August 2013 - 03:53 AM, said:

Look, read up on Elo rating. It's made for chess, where it's 1v1, strict playing rules, unchanged playing environment and the player solely relies on himself to win.

It works amazingly well for teams of random players actually. When looking at all possible matchups, good players are in more winning matches then bad players. You can program your own simulation for it if you're into it but here is a short explanation example:

Imagine four player, A,B,C,D playing a game of tug of war. The players have a strength of 1,2,2,3 each.

The possible match-ups are:

AB | CD = 3 | 5
AC | BD = 3 | 5
AD | BC = 2 | 2

When both sides are equally strong each has a 50% of winning.

Player A wins 16% of his games.
Player B wins 50% of his games.
Player C wins 50% of his games.
Player D wins 83% of his games.

Now if you apply Elo twice to the matches in order they are listed above, starting all players at 1300 Elo:

A(1275) B(1275) | C(1325) D(1325)
A(1250) C(1300) | B(1300) D(1350)

If won by left side: A(1275) D(1375) | B(1275) C(1275)

A(1275) B(1275) | C(1275) D(1375) no change because right team is expected to win and wins.

A(1275) C(1275) | B(1275) D(1375) no change because right team is expected to win and wins.

If won by left side: A(1275) D(1375) | B(1275) C(1275) no change because left team is expected to win and wins.

If won by right side: A(1254) D(1354) | B(1308) C(1308)

If won by right side: A(1225) D(1325) | B(1325) C(1325)

A(1225) B(1325) | C(1325) D(1325) no change because right team is expected to win and wins.

A(1225) C(1325) | B(1325) D(1325) no change because right team is expected to win and wins.

If won by left side: A(1253) D(1353) | C(1297) B(1297)

f won by right side: A(1225) D(1325) | C(1325) B(1325) no change because right team is expected to win and wins.

The possible outcomes are now

A(1275) B(1275) C(1275) D(1375)
A(1254) B(1308) C(1308) D(1354)
A(1253) B(1297) C(1297) D(1353)
A(1225) B(1325) C(1325) D(1325)

The second and third result have the players nicely shorted out. The first and the last one don't make a difference yet because based on the evidence (the matches) there is no reason to assume B and C are better then A or worse then C. Another round through the system will reduce the chances of this happening 50% for each round.

Now this is a simple system, with a simple mechanism to compute who wins, but that is besides the point. It shows that Elo works fine when comparing teams.

----

What you are probably experiencing are problems with the match maker being unable to find enough players near your skill level. This definitely needs tweaking, but its not Elo.

Edited by Hauser, 16 August 2013 - 05:43 AM.

[DeadlyNerd]

Hauser, on 16 August 2013 - 05:42 AM, said:

-snip-

First, I said matchmaker needs scrapping, not Elo. It's a proven 1v1 player rating system...

Second, you can calculate all you want, but Elo is not made for pairing up groups of individual players because of 1 basic principle. 50% chance doesn't necessarily mean that every second try will be a hit. Over a longer period of time a the hit rate might seem 1 out of 2, but when taking short intervals it's not.
Combine that with the fact that the current matchmaker takes different combinations of these ratings every time, you'll very rarely get a balanced matchup.
No calculation can prove current matchmaker works fine because no calculation is, or can be practically, done on a large scale.

This was the theoretical part.

Practical part generally disqualifies any theoretical calculations right off the bat. Aside from that, as Fabe stated, what if bad players are carried by good teams too often and then end up in high tiers.
Sure, they'll drop in rating with time, but so will those that lose a long side them and the whole process starts over.
Using Elo rating for multiple player matchups is a bad concept and, if not tweaked to the point where it doesn't resemble Elo at all, will keep causing problems.

Edited by DeadlyNerd, 16 August 2013 - 06:12 AM.

[Hauser]

DeadlyNerd, on 16 August 2013 - 06:05 AM, said:

First, I said matchmaker needs scrapping, not Elo. It's a proven 1v1 player rating system...

Actually from your first post say that Elo can not be applied to anything but 1v1. This is pertinently wrong. As I just showed, it works just fine for teams.

DeadlyNerd, on 16 August 2013 - 06:05 AM, said:

Second, you can calculate all you want, but Elo is not made for pairing up groups of individual players because of 1 basic principle. 50% chance doesn't necessarily mean that every second try will be a hit. Over a longer period of time a the hit rate might seem 1 out of 2, but when taking short intervals it's not.

Combine that with the fact that the current matchmaker takes different combinations of these ratings every time, you'll very rarely get a balanced matchup.
No calculation can prove current matchmaker works fine because no calculation is, or can be practically, done on a large scale.

This was the theoretical part.

I wrote down the whole possibility tree for those 6 matches. I have described every possible outcome in this system for the first 6 matches. There are no other possible outcomes.

If you repeat this a few more times you'll see that on the next branch you'll have 8 outcomes, 6 of which provide the correct ordering. On the next iteration you'll have 16 outcomes, 14 of which provide the correct ordering. On those two branches that don't provide the correct ordering there is no empirically observable difference between either A, B and C or B, C and D. They deserve the same rating.

It may not provide the correct result on the short run, but that is to be expected for any measuring system. Without a certain amount of comparison you can not actually rank anybody properly. To be exact, to rate a system of n players, you need at least n log n comparisons.

I'm not talking about balance here and I haven't really gone into how the matchups are assembled. Most matches I showed are decidedly unbalanced. But for the argument you make that is not necessary. If you want to talk about how the match maker is creating unbalanced matches you are welcome too, but you should be clear about that you are talking about unbalanced matches. Right now it seems you are complaining about Elo.

DeadlyNerd, on 16 August 2013 - 06:05 AM, said:

Practical part generally disqualifies any theoretical calculations right off the bat. Aside from that, as Fabe stated, what if bad players are carried by good teams too often and then end up in high tiers.
Sure, they'll drop in rating with time, but so will those that lose a long side them and the whole process starts over.
Using Elo rating for multiple player matchups is a bad concept and, if not tweaked to the point where it doesn't resemble Elo at all, will keep causing problems.

Yes. A bad player can be carried by a good team. But let me break that down in two situations.

If the bad player is playing with 3 friends that are good enough to cary him. In that case Elo is no longer relevant as a personal rating. It simply reflects the rating of the group of 4 players.

If a bad player by sheer luck always ends up on the winning team he can indeed end up with inflated Elo. Yet the chance that this happens, the chance that a single player always ends up on the winning team without actually contributing to the win are very small. You can see this in the example I gave, the worst player is A who ends up winning only 16% of his games.

The best of Elo is that if a player with an inflated Elo isn't able to win a game he is expected to win, the system will take it as strong evidence that he has been over rated and remove a significant amount of his score, quickly setting him back to his proper rating. So even if you have a lucky streak, you'll be put into place at the first moment you don't live up to it.

[IceSerpent]

Hauser, on 16 August 2013 - 05:42 AM, said:

Imagine four player, A,B,C,D playing a game of tug of war. The players have a strength of 1,2,2,3 each.

The possible match-ups are:

AB | CD = 3 | 5
AC | BD = 3 | 5
AD | BC = 2 | 2

When both sides are equally strong each has a 50% of winning.

Hauser, this only works for things like tug of war where players' "abilities" (physical strength in this case) are added together. In MWO the AD team will almost always lose to BC team because player A is completely useless and player D is unable to take out the whole team of average players B and C singlehandedly.
This is pretty much the root of the current problem with Elo matching - MM should never, ever put players with vastly different ratings on the same team.

Edited by IceSerpent, 16 August 2013 - 07:18 AM.

[Hauser]

IceSerpent, on 16 August 2013 - 07:17 AM, said:

Hauser, this only works for things like tug of war where players' "abilities" (physical strength in this case) are added together. In MWO the AD team will almost always lose to BC team because player A is completely useless and player D is unable to take out the whole team of average players B and C singlehandedly.

The actual mechanism is irrelevant. It's only there so I can run a simulation with consistent results. It's also not an example of good match making, it is terrible. It's just there to show that even with the influence of other players you can run a rating system for individual players. As long as your individual performance has some influence on the outcome you can use Elo. It just takes a while to converge.

IceSerpent, on 16 August 2013 - 07:17 AM, said:

This is pretty much the root of the current problem with Elo matching - MM should never, ever put players with vastly different ratings on the same team.

The match maker is putting people with significant difference in Elo on the same team. But that is not what it sets out to do. It tries to find 24 players near a given value. As time goes by and those players aren't available it will start accepting players that are further and further from the target value.

Since this isn't working it appears that there are not enough players around a certain Elo online at the same time. This can either be a complete lack of the right players or the players can be sucked into other less ideal matches just to fill them out.

http://mwomercs.com/...-making-update/

Quote

How does the match maker compose a teams Elo rating, is it average rating or closest to a target?

It's closest to a target value, so the match maker starts trying to make a match for an Elo of say 1300 and will pull in players to those teams closest to those values; however, as mentioned earlier within growing thresholds and those curves will be tuned. Currently it may be a bit 'sloppy' about how it's filling those buckets but over time it will be tuned to be much more precise.

We need to do this carefully over time as generally the cost of precision is time to find a match we want to slowly find a very nice balance between time to find a match and the number of matches that are correctly composed.

[DeadlyNerd]

Hauser, on 16 August 2013 - 07:16 AM, said:

Actually from your first post say that Elo can not be applied to anything but 1v1. This is pertinently wrong. As I just showed, it works just fine for teams.

Actually it's pretty correct because, as even PGI stated, Elo needs a larger playerbase to be efficient, thus making your small scale calculations irrelevant.

Hauser, on 16 August 2013 - 07:16 AM, said:

I wrote down the whole possibility tree for those 6 matches. I have described every possible outcome in this system for the first 6 matches. There are no other possible outcomes.

If you repeat this a few more times you'll see that on the next branch you'll have 8 outcomes, 6 of which provide the correct ordering. On the next iteration you'll have 16 outcomes, 14 of which provide the correct ordering. On those two branches that don't provide the correct ordering there is no empirically observable difference between either A, B and C or B, C and D. They deserve the same rating.

First, MWO has over 1k players, not 4. Second, they wont necessarily play against each other in a full circle, player 1 might never play against player 1000. Third, who says they need to play in exactly that order.
You also forgot a "couple" matchups.

Hauser, on 16 August 2013 - 07:16 AM, said:

It may not provide the correct result on the short run, but that is to be expected for any measuring system. Without a certain amount of comparison you can not actually rank anybody properly. To be exact, to rate a system of n players, you need at least n log n comparisons.

I'm not talking about balance here and I haven't really gone into how the matchups are assembled. Most matches I showed are decidedly unbalanced. But for the argument you make that is not necessary. If you want to talk about how the match maker is creating unbalanced matches you are welcome too, but you should be clear about that you are talking about unbalanced matches. Right now it seems you are complaining about Elo.

A measuring system that doesn't provide a correct result when tested isn't a well calibrated, or even functional, measuring system.

Again, I think I've been very clear on this. Elo is not the problem, it's how this matchmaker is using it to rank players. Elo is not made for anything more than 1v1 and no system using Elo, to rank players within randomly assembled groups, will function properly.

Hauser, on 16 August 2013 - 07:16 AM, said:

Yes. A bad player can be carried by a good team. But let me break that down in two situations.

If the bad player is playing with 3 friends that are good enough to cary him. In that case Elo is no longer relevant as a personal rating. It simply reflects the rating of the group of 4 players.

If a bad player by sheer luck always ends up on the winning team he can indeed end up with inflated Elo. Yet the chance that this happens, the chance that a single player always ends up on the winning team without actually contributing to the win are very small. You can see this in the example I gave, the worst player is A who ends up winning only 16% of his games.

There is no sheer luck when the system itself already is loosening constraints on Elo rating when assembling groups. Low elo rating players will most definitely end up in a winning team.

Hauser, on 16 August 2013 - 07:16 AM, said:

The best of Elo is that if a player with an inflated Elo isn't able to win a game he is expected to win, the system will take it as strong evidence that he has been over rated and remove a significant amount of his score, quickly setting him back to his proper rating. So even if you have a lucky streak, you'll be put into place at the first moment you don't live up to it.

>>a player<< You're talking singular here. That pretty much negates your whole argument as you began changing rules to benefit it.
Again, that lucky streak is not your personal accomplishment, but your team's, if you're a bad player.

IceSerpent, on 16 August 2013 - 07:17 AM, said:

Hauser, this only works for things like tug of war where players' "abilities" (physical strength in this case) are added together. In MWO the AD team will almost always lose to BC team because player A is completely useless and player D is unable to take out the whole team of average players B and C singlehandedly.
This is pretty much the root of the current problem with Elo matching - MM should never, ever put players with vastly different ratings on the same team.

THANK YOU.
I had doubts anyone would actually understand.

Edited by DeadlyNerd, 16 August 2013 - 10:37 AM.

[Tehtos]

You don't want low skill people on the same team as high skill people, I agree. Tonnage-matching does not care about skill level. Guess what happens when only tonnage is used to create matches? Wait, we don't have to guess, we can infer from the old tonnage-class matching system. It was horrible for most people.

While I think the current match-maker could use some tweaking (and a larger pool), a tonnage only system would be worse.

[GingerBang]

DeadlyNerd, on 16 August 2013 - 12:53 AM, said:

Srsly, it's an invalid concept in applying Elo rating in anything more than 1v1, and there are more bandaids on it than on an anime patient.

Scrap the matchmaker already and just give us tonnage restrictions, it's taking too long to find a match and there's really no benefit.

Screw tonnage restrictions. That will only amplify the maximum damage per ton meta game. We need real, comprehensive battlevalues. That way an awesome with 8 ML's is not rated equally to an awesome with 4 ER-PPC's and a tiny little engine

[IceSerpent]

Hauser, on 16 August 2013 - 10:27 AM, said:

The actual mechanism is irrelevant.

On the contrary, it's extremely relevant - when a team tries to lift something heavy your scheme works, as one very strong person + one very weak person is the same as two average people. The effectiveness of the team is a straight up adition of their strengths.
In MWO you can't "add" player skills like that - weak players are simply useless and contribute nothing to the team. So, 1 weak player + 1 strong player vs. 2 average players is the same as 1 strong player alone vs. 2 average players.

Quote

The match maker is putting people with significant difference in Elo on the same team. But that is not what it sets out to do.

You are missing the point - it doesn't matter what it's trying to do, people with significant Elo difference should not be on the same team, no matter what. It's better to sit in queue for 15 mins than to get team makeups like we are getting now.

[Screech]

Why are people acting like tonnage limits are not coming? It has been stated now a couple times. They are placing tonnage limits on groups, remove group size limits and then match from group size and Elo score. What the tonnage limits are is unknown, but the rest has been stated as the plan.

They have decided to remove arbitrary group limits and replace with arbitrary weight limits. Lesser of two evils, will improve the matchmaker.

[DeadlyNerd]

Tehtos, on 16 August 2013 - 01:13 PM, said:

You don't want low skill people on the same team as high skill people, I agree. Tonnage-matching does not care about skill level. Guess what happens when only tonnage is used to create matches? Wait, we don't have to guess, we can infer from the old tonnage-class matching system. It was horrible for most people.

While I think the current match-maker could use some tweaking (and a larger pool), a tonnage only system would be worse.

Actually what was in the game was a mech weight tier matcher, not tonnage matcher. That meant that an Atlas on 1 team could be an Awesome on the other and that sucked.

[IceLom]

IceSerpent, on 16 August 2013 - 01:26 PM, said:

On the contrary, it's extremely relevant - when a team tries to lift something heavy your scheme works, as one very strong person + one very weak person is the same as two average people. The effectiveness of the team is a straight up adition of their strengths.
In MWO you can't "add" player skills like that - weak players are simply useless and contribute nothing to the team. So, 1 weak player + 1 strong player vs. 2 average players is the same as 1 strong player alone vs. 2 average players.

You just proved why he is correct...

Because in that context of what players are available, the most balanced team is the strong + weak vs the 2x middle. Sure the one team might still be allot stronger but its the most fair possible combination that can be created given the available players.

[Jestun]

"completely random skill levels" is not better than "a WIP ELO-based matchmaker with the rules tweaked as required".

I'm not sure the playerpool can support even loose weight criteria while offering balanced matches, especially not when you account for the addition of:

1st person queues vs. 3rd person queues
regional account system splitting the playerbase
quick match vs. community warfare

Plus the already existing assault vs. conquest and PUG vs pre-made splits.

[Hauser]

IceSerpent, on 16 August 2013 - 01:26 PM, said:

You are missing the point - it doesn't matter what it's trying to do, people with significant Elo difference should not be on the same team, no matter what. It's better to sit in queue for 15 mins than to get team makeups like we are getting now.

Yes. Like I said before, I agree. People with significant Elo difference should not be on the same team. However that was not the point the OP was making.

[Jman5]

People really need to stop thinking that a good match maker will result in every game being good. Rolls are common even among evenly matched players. I can have a great game in my Atlas with 4 kills/800 damage, then completely blow it the next game, which sets off a series of events that cause my team to get 12-0'd.

People are not robots who consistently perform the same every game and on every map.

I remember what it was like before Elo was introduced. 90% of the people I fought were terrible and couldn't aim. It didn't matter if he was twice my weight because I was generally twice as experienced with the game. A decent 4-man just mopped the floor without even trying. The odds of them rolling another equally skilled 4-man was low.

[IceSerpent]

IceLom, on 16 August 2013 - 02:22 PM, said:

You just proved why he is correct...

Because in that context of what players are available, the most balanced team is the strong + weak vs the 2x middle. Sure the one team might still be allot stronger but its the most fair possible combination that can be created given the available players.

Are you trolling for a faceplam meme or seriously don't understand why it's nowhere near being most fair?

[Tehtos]

DeadlyNerd, on 16 August 2013 - 02:10 PM, said:

Actually what was in the game was a mech weight tier matcher, not tonnage matcher. That meant that an Atlas on 1 team could be an Awesome on the other and that sucked.

Yes I understand what we had in closed was tonnage class (or weight tier as you call it), hence why it must be inferred.

The point is that matching based on tonnage alone will still allow two players with vastly different skills to be matched against each other. The goal is to match people with similar skills, so people with low skills do not get on the same team as people with high skills.

[Mackman]

Fabe, on 16 August 2013 - 05:33 AM, said:

Another problem with the Elo is that if I recall correctly it only looks at whether or not you were on the winning team which isn't a real good indication of individual skill. Put a poor player like me on a team good enough to carry me and even if I don't actually improve my Elo score will still go up just for being on the winning team.

But it's extremely unlikely (mathmetically speaking) that you winning despite your low skill would happen more often than you losing because of it. It's simple probability: Given a sufficiently large number of matches, you will eventually be placed with people of the same skill (or "effectiveness", if you bring in people who are intentionally gimping themselves).

Elo, if properly implemented, will work: That's why League of Legends has been using it successfully ever since it started out (you know, before it became the biggest game in the world by using a good Elo matchmaking system.).