263 replies to this topic

[KodiakGW]

All I'm going to say is one word.

Vindicated.

They did exactly what I said should be done. Base matchmaker on skill, not just win/loss. Carried too many poor players. All those threads arguing that point paid off.

[oneproduct]

IraqiWalker, on 17 August 2015 - 04:27 PM, said:

Actually, Win loss has always been a bad way of measuring a player's performance. Many here can relate to this experience:

I clocked in 1200 damage, 4 kills, and about as many assists, yet the team still lost. Why is it that my Elo drops then, when I clearly performed well above the skill tier of most players in the game. I singlehandedly out-damaged our entire Charlie Lance combined, and wiped out an entire enemy lance as well, and I wasn't in an LRM mech, btw.

Yet using Elo, I would end up losing points. Because this is not an absolutely static 1 v 1 game, where we both have the exact same pieces, and the exact same number of them too. My Com-2D is not the same as my enemy's Com-2D, we have different engines, and different piloting skills, and preferences.

Since this game's release, we've discussed how Elo is not a good way to match make MWO.

Namely because you have 23 other players (11 of which on your own team) whose behavior is outside of your control.

Here's another example where Elo fails, but the PSR would stand a chance:

We drop into a match, according to Elo, my team is slated to win, and so if we lose we will also lose Elo score. If we win, our scores don't increase.

The entirety of Bravo lance decides it's more fun to teamkill each other, or suicide, or disconnect, or whatever else, that puts us at an immediate disadvantage. Despite good play, we still lose, and our Elo Score drops. How is that an accurate representation of our skill?

At least in LoL, you can kinda use Elo, because the champions are identical, and the teams are FIXED. Only me and my other 4 friends can be on this team. So Elo can kinda be applied. That's not the case here.

By the way, LoL no longer uses Elo, as you said. They stopped using it halfway through season 2, because of these exact same problems. Instead we now have the League Point system. Which is incredibly more accurate, and fairer. It's also beyond a modification to Elo, unless by modification you mean "re-write of the entire system".

Here's another example, that is all too common with Elo: We get 1 or 2 players that are clearly way out of their league. They can't even move the mechs yet. Our team still managed to roflstomp the opposition, and even though those two players dealt all of 2 points of damage between the two of them. Their Elo will rise. Causing them to face even tougher opponents, when they clearly don't have the skill to be there.


Better yet. We drop into match, and I accidentally die early. 0 damage. 0 kills. 0 assists. Team wins. Why should my Elo increase, when my team won DESPITE me being there, instead of with/because of me?

The PSR will at least help a bit with mitigating those problems. Give it a few months, and it can work much better than Elo does right now.

In my post you quoted I readily said that LoL does not use Elo anymore, but a system that is heavily based on Elo. It has the same concept of only judging you on win/loss and making you lose an amount of points based on your rank relative to others. The only thing different that they do, from what I could gather, is that they rate you based on the average ranking of people in your bracket rather than the skill of people in your particular match.

Your example of when you get a huge match score in a match and still lose ranking points is also not relevant. Statistics will average this out. The number of games where you have a huge match score and lose will be low compared to the number of games where you have a huge match score and win. Also you will have games that you win and have a low match score.

Match score is also not indicative of how well you do. For example, imagine if the most important thing in the game was putting up a UAV but the match score for UAVs was extremely low. You could have a 100% win rate because you constantly manage to get good UAVs up, but your PSR may be terrible because you don't get a lot of match score for it.

PSR is too arbitrary. Win/loss is not due to such concepts as https://en.wikipedia...f_large_numbers

Please describe to me what the League Point system in LoL does that you believe is so vastly different than Elo that makes it work. Don't get me wrong, I do believe that it has the ability to work better, as does any system that you tend to specialize for your particular case, so long as the basis is sound. In the case of LoL, win/loss is still the basis, so it is still objective in at least that respect. And if it does work, we could simply try to employ the same thing here, using just brackets and win/loss, rather than relying on something arbitrary like match score.

Also to address your point about not coming into the game with equivalent gear, that is a sad reality, but not something that a matchmaker can account for. Firstly, it is difficult or impossible to determine which gear is better, along the lines of battle value. Second, a premise of these kind of ranking systems is that you will do your utmost to win. If you are not trying your best to win, then the matchmaker can't possibly know that and even if it did, what would it do? Decide not to add or subtract any rank points for that match? That is what unranked games (which we don't have in MWO at the moment) are typically for.

Edited by oneproduct, 17 August 2015 - 04:44 PM.

[JonahGrimm]

I haven't read this entire thread, but do want to add a voice - Player Rank should never be allowed to be public.

I know, but hear me out:

There are people who live for player rank, and there are those that don't. If player rank becomes a thing this community cares about as a whole, people who try odd builds, do interesting things, or even stretch to a new chassis will be penalized for doing so on the social side of the fence.

Sure, i'm hooked on shadow cats right now - and I'm getting pretty good at my cat of choice. My skills in my locusts, though, have atrophied. If this community gets it in its head that Player Rank 1 has real significance, then when I switch back to those locusts, or go to master hunchbacks, or try to figure out assaults, my Rank will suffer - and the community, then, will react to that.

I've got no intention of changing my playstyle; i have fun, and that's what I'm here to do. I can see dozens of scenarios, though, where player rank becomes an anti-fun metric.

If we must display rank, then make that a one-way trip: once you've achieved a rank above four, you permanently gain some sort of marker, title, or sparkly avatar thing that you keep regardless of what happens to your rank in the future. If I'll self-correct to rank 3, let's say, when learning a new mech... that's fine, but there shouldn't be significant pressure to maintain a ranking at all costs, even for the veterans.

That way lies a certain amount of stagnation and animus.

[bad arcade kitty]

Gas Guzzler, on 17 August 2015 - 04:06 PM, said:

It seems that the old system was more about trying to game you to have a 50/50 W/L ratio

any mm tries to give you matches vs decent opponents

hint, decent opponents it's those who can win against you... roughly in a half of the cases

see how 50/50 appears?

[IraqiWalker]

oneproduct, on 17 August 2015 - 04:35 PM, said:

In my post you quoted I readily said that LoL does not use Elo anymore, but a system that is heavily based on Elo. It has the same concept of only judging you on win loss and making you lose an amount of points based on your rank relative to others. The only thing different that they do, from what I could gather, is that they rate you based on the average ranking of people in your bracket rather than the skill of people in your particular match.

Your example of when you get a huge match score in a match and still lose ranking points is also not relevant. Statistics will average this out. The number of games where you have a huge match score and lose will be low compared to the number of games where you have a low match score and lose. Also you will have games that you win and have a low match score.

Match score is also not indicative of how well you do. For example, imagine if the most important thing in the game was putting up a UAV but the match score for UAVs was extremely low. You could have a 100% win rate because you constantly manage to get good UAVs up, but your PSR may be terrible because you don't get a lot of match score for it.

PSR is too arbitrary. Win/loss is not due to such concepts as https://en.wikipedia...f_large_numbers

LP actually accounts for a lot of elements, not just win loss, in fact, kills, assists, gold, minions, item purchases, wards placed ... etc. Those are all factored in.

The UAV example you used isn't that accurate, because all those spotting assists from UAVs, plus scouting bonuses, plus any other goodies from the UAV do help give you a better PSR than you would, if you weren't using PSR, or using Elo.

Also, do note that I did put "By the way, LoL no longer uses Elo, as you said." acknowledging that you stated that.

Also, using a hypothetical impossibility like that UAV example isn't a good point. Why would something considered the "most important thing in the game" not have a match score that corroborates that fact?

A better one would be if I put the UAV up, and then died, I would have a better match score than 0, but that's a better representation than Elo, because at least I bothered to put a UAV up, in a good spot, and unless we lose, I will receive some reward. Whereas Elo doesn't care. I could still die instantly, without putting a UAV up, and it wouldn't matter to Elo.

So far, I have not seen a single competitive game, with this much variety, that uses Elo, and it works out well (no, even in CS, there are many discussions about Elo being useless for anything short of competitive team play. Competitive solo play has MANY problems with Elo. Most are identical to what we have here.)

[Piney II]

JonahGrimm, on 17 August 2015 - 04:40 PM, said:

I haven't read this entire thread, but do want to add a voice - Player Rank should never be allowed to be public.

I agree with this. The epeen measuring will be off the hook here, and unless you're a top dog, you will be discounted as a "mouth breather / steering wheel underhive".

Nothing like further dividing the player base...................

The flip side is some of these self professed elite players will be faced with the fact that they're not as leet as they think.

Keep the rankings private. We don't need epeen measuring and bruised egos.

[oneproduct]

bad arcade kitty, on 17 August 2015 - 04:44 PM, said:

any mm tries to give you matches vs decent opponents

hint, decent opponents it's those who can win against you... roughly in a half of the cases

see how 50/50 appears?

Thank you for being a voice of reason bad arcade kitty. And to Gas Guzzler, I would like to ask you what more could you want from a matchmaking system than to have a 50/50 W/L ratio?

Why should you have a 60/40 W/L and have more fun than someone else who gets stuck with a 40/60 W/L ratio? Who has to suffer for your pleasure? If you are matched with opponents of your skill, you will have a 50/50 W/L ratio. Otherwise if you are really skilled and there simply aren't any opponents of your skill level, you will start going above 50/50, but in this case it would not be the matchmaker's fault.

[oneproduct]

IraqiWalker, on 17 August 2015 - 04:44 PM, said:

LP actually accounts for a lot of elements, not just win loss, in fact, kills, assists, gold, minions, item purchases, wards placed ... etc. Those are all factored in.

Also, using a hypothetical impossibility like that UAV example isn't a good point. Why would something considered the "most important thing in the game" not have a match score that corroborates that fact?

Can you provide some source referring to LoL using something other than just win/loss for ranking? I can't find anything that references that, or much of the details at all. I can only find a few things like this which I can't be sure are reliable or not, but suggest that win/loss is the only factor: https://boosteria.or...-of-legends-mmr

Here it suggests that things like kills, assists and deaths are not factored in: http://forums.na.lea...d.php?t=3245063

The hypothetical impossibility isn't that impossible. The reason why it is possible is because the match score for actions is arbitrary. It could be that UAVs are very important, but PGI does not give you the right amount of match score to reflect that.

In a hypothetical future when LRMs are super powerful due to a balance patch, suddenly TAG assist could be the most important thing in the game, but if they didn't update their PSR scoring system it wouldn't reflect that.

The key thing is that match score for action is arbitrarily defined as opposed to win/loss which is objective.

The problem with PSR is that you are at the mercy of the scoring system. Someone who wins 100% of their games by doing genuinely useful things could have a lower PSR than someone with a 50% win rate who does less useful things, because the actions they did were not rewarded according to their level of merit. And the same problem occurs here: it is difficult to determine the merit of actions.

As an exercise, if I asked you to give a match score value to every action in the game and it were used to calculate everyone's PSR, would you be confident you could do it? How often would you need to revise it?

Also there is the problem of being able to game the system. You earn more points for doing more damage to a mech rather than killing it efficiently. Headshotting a mech instantly would be worth less than breaking both the arms and one of the legs of a mech with all torso mounted weapons.

It encourages you to squeeze out every point from a match, whereas with win/loss there is nothing to squeeze.

Edited by oneproduct, 17 August 2015 - 05:04 PM.

[Throat Punch]

All i want to know is when i get to whip out my epeen so you can all laugh at how small it is.

[IraqiWalker]

oneproduct, on 17 August 2015 - 04:48 PM, said:

Thank you for being a voice of reason bad arcade kitty. And to Gas Guzzler, I would like to ask you what more could you want from a matchmaking system than to have a 50/50 W/L ratio?

Why should you have a 60/40 W/L and have more fun than someone else who gets stuck with a 40/60 W/L ratio? Who has to suffer for your pleasure? If you are matched with opponents of your skill, you will have a 50/50 W/L ratio. Otherwise if you are really skilled and there simply aren't any opponents of your skill level, you will start going above 50/50, but in this case it would not be the matchmaker's fault.

But Elo doesn't provide a 50/50 chance here. It pits a team with 12 average pilots, against a team with 6 god tier pilots, each worth at least 3 average players, and 6 crappy pilots. Technically, both teams have close enough Elo, possibly identical. Yet that match is not fair in the slightest, or even close to 50/50 I'd say it's a 90% chance the team with the 6 god tier pilots will win without breaking a sweat.

I don't see the problem here. One system only counted wins and losses, which for the overwhelming majority of the time, were outside of the player's control. It assigned you a score regardless of how good, or bad you were.

The other focuses on your own performance, to better place you in teams closer to you in performance. Allowing for a significantly better chance of getting that fabled 50/50 match.

oneproduct, on 17 August 2015 - 04:53 PM, said:

Can you provide some source referring to LoL using something other than just win/loss for ranking? I can't find anything that references that, or much of the details at all. I can only find a few things like this which I can't be sure are reliable or not, but suggest that win/loss is the only factor: https://boosteria.or...-of-legends-mmr

The hypothetical impossibility isn't that impossible. The reason why it is possible is because the match score for actions is arbitrary. It could be that UAVs are very important, but PGI does not give you the right amount of match score to reflect that.

In a hypothetical future when LRMs are super powerful due to a balance patch, suddenly TAG assist could be the most important thing in the game, but if they didn't update their PSR scoring system it wouldn't reflect that.

The key thing is that match score for action is arbitrarily defined as opposed to win/loss which is objective.

The problem with PSR is that you are at the mercy of the scoring system. Someone who wins 100% of their games by doing genuinely useful things could have a low PSR because the actions they did were not rewarded according to their level of merit.

MMR is a secondary formula, on top of LP. Other than the original announcement, and a few forum posts from the devs, they keep a tight lip on the inner workings of their ranking system. I'll see if I can find more information, but I highly doubt it.

I don't see that as arbitrarily defined. An action's importance is reflected in it's match score. If an action brings too little to make a difference, on the score, people will steer away from it, and the ones that give you a better match score become the ones focused on.

I also don't think PGI is made up of lobotomized techs. They added in more rewards geared towards helping light mechs, a weight class that has long suffered from craptastic rewards, despite all the risk taken. So it seems they know what they're doing. Even if it's not fully.

I highly doubt they would reward something silly with more match score than something that's actually important.

On the other hand, do remember their formula. You are not going to be doing any one thing in the match, and that's it. You match score is a combination of all actions you make. This means that even in what are realistically impossible conditions, you can still get a high match score, IF YOU ARE GOOD. Whereas Elo doesn't care, and will bump you up or down, regardless of your skill.

Elo was a system designed specifically for measuring skill, when it fails at doing that for a game, then it needs to be replaced. This is something that has been talked about in the industry for years now. Many games refrain from using Elo for this reason. Some either use a ladder system with their own scoring method, or no MM instead of using Elo.

How is Elo good, when it puts tier 5 players against tier 1 players, in the same match?

In this case, Elo score is very arbitrary, because winning and losing is about random luck, rather than player control. Which is anathema to Elo, a system designed for a game with absolute and rigid control over every single detail (chess).

[Mechwarrior Buddah]

KraftySOT, on 17 August 2015 - 04:16 PM, said:

Unless they dramatically rebalance these mechs and quirks.

I thought thats what the combat upgrade rebalancing was supposed to be XD

[AssaultPig]

you're never going to get a matchmaking system that evaluates every player's likely contribution perfectly, but this seems likely to do a much better job than simple W/L elo.

The only thing that really worries me about it is the lack of release valves; it can already take 3-4 minutes for me to find a game at offpeak times, and lengthening that would suck.

On the other hand it'll be nice not to get dropped with/against obvious new players who don't have a handle on basic stuff like twisting or range bands.

also, elo isn't a measure of skill, it's a measure of win expectancy. The two are more or less interchangeable in 1v1 games (i.e. chess), less so in a game like this

Edited by AssaultPig, 17 August 2015 - 05:12 PM.

[bad arcade kitty]

you mistake elo and matchmaking, mm it's a way to apply elo

the tier system definitely can help the matchmaking

meantime punishing/rewarding people for their score instead of the pure win/loss elo will cause problems and it is generally considered worse than the pure elo

[Scandinavian Jawbreaker]

Piney, on 17 August 2015 - 04:47 PM, said:

I agree with this. The epeen measuring will be off the hook here, and unless you're a top dog, you will be discounted as a "mouth breather / steering wheel underhive".

Nothing like further dividing the player base...................

The flip side is some of these self professed elite players will be faced with the fact that they're not as leet as they think.

Keep the rankings private. We don't need epeen measuring and bruised egos.

Well it wouldn't be self professed anymore and the case might actually be that they know what they're talking about. Some players drive good players away from the forums because of their... Different ideas of gameplay they seem to defend very aggressively while some have no idea what it is in the higher levels of play.

I'm not really that interested in epeen or waving it. I just want some quality to the balance discussions around here.

And now I sound like a jerk, but hey, deal with it.

[Piney II]

AssaultPig, on 17 August 2015 - 05:11 PM, said:

you're never going to get a matchmaking system that evaluates every player's likely contribution perfectly, but this seems likely to do a much better job than simple W/L elo.

The only thing that really worries me about it is the lack of release valves; it can already take 3-4 minutes for me to find a game at offpeak times, and lengthening that would suck.

Agree. This move sounds like it's in the right direction. The wait times are a concern. I imagine PGI has built in release valves. We'll see soon.

[oneproduct]

@IraqiWalker
(going to stop quoting posts, as they're getting too big now )

In your two examples of teams having mismatched skilled players, it's not the fault of the matchmaker, but of the limited population.

The matchmaker tries to (or should) limit Elo variance, so if possible, it will try to match 12 good vs 12 good. However, when that's not possible, it is better to have 12 average vs 6 good and 6 bad than to have 12 average vs 6 good and 6 average so at least you have some hope of somewhat evening things out. Even then, it won't (or shouldn't) try to pick people who are so dramatically different in rating. They had a post about the average variance between the top and bottom ranked players in matches. I believe for single queue it was below 50 and for group queue it was around 200, which is likely due to the fact that it can't split up groups to try to put an equal number of good players on each team.

LP is just the visible "currency" you get to track your progress between one bracket to the next. The amount of LP you gain is directly correlated to your MMR and the amount of MMR you gain or lose as the result of a match being won or loss. Here is the reference and a quote: http://leagueoflegen...i/League_system

Quote

League Points

Each ranked game you win earns you League Points while each lost one decreases your current LP. The exact amount of lost or earned League Points per game depends on your hidden Match Making Rating. If you have higher MMR than a set amount for your division you gain more LP per won game and lose less LP per lost game, if your MMR is lower you gain less and lose more LP per game.

LP and MMR uses the same basis as Elo. Again, the only difference I could find is that it compares your rating vs the average for your division rather than against the rating of people in your particular match.

About the arbitrary nature of match score, I will concede this: if you award the perfect amount of match score for each action based on the actual merit of that action, then it could work well. However, even at the moment, if both teams simply stood AFK next to each other for a long time at the start of the match, they would get a lot of "lance in formation" awards even though they aren't doing anything useful. However at some points, keeping lance formation is actually useful.

Edited by oneproduct, 17 August 2015 - 05:22 PM.

[Goose]

I appalled to see how many people blame Elo for Teh Devs use of averaging and release valves …

[Piney II]

ugrakarma, on 17 August 2015 - 05:15 PM, said:

Well it wouldn't be self professed anymore and the case might actually be that they know what they're talking about. Some players drive good players away from the forums because of their... Different ideas of gameplay they seem to defend very aggressively while some have no idea what it is in the higher levels of play.

I'm not really that interested in epeen or waving it. I just want some quality to the balance discussions around here.

And now I sound like a jerk, but hey, deal with it.

Most of us are not at the higher levels of play. So that disqualifies us from any balance discussions (like it matters here).....the unwashed masses?

I think even lower ranked players just might have some good input, but that would be ignored by the top dogs offhand.

We don't need that sort of exclusion here.

[InspectorG]

" Tier 5 being comprised of pilots who are in dire need of additional training. "

Oh my...many will be surprised to be relegated to this tier.

The Salt must flow.

[Davers]

I'm not sure how big a difference this will make. It's still a team game, still 12v12. If people think their scores aren't totally affected by their teammates, they are dead wrong. But at least top players won't face new/bottom players anymore. Or at least until they add release valves because the top players can't find matches.