51 replies to this topic

[MrMadguy]

As you know, I have always had some suspicions, that our PSR MM is broken.

Everybody know this picture, created by me:


It's based on this picture, provided by devs themselves, with actual MS values, discovered by players, added to it:


Everybody knows, that according to this picture PSR system is biased towards increasing of rating. Initial bias comes from simple fact, that rating decrease below 250MS is smaller, than rating increase above it:



Yeah. Problem is - that's not all. Now I have new suspicion - that even with bias towards increasing my stats should have been enough to drop to Tier 4.

So, I've created simple PSR simulator and entered my performance parameters to it:


There are three parameters, we don't know - it's actual values of "small", "moderate" and "large" PSR changes. So, I had to play with this parameters to try to make my rating increase with my stats. I started with 1, 2, 3. And the most simple approach to achieve this goal - to increase "small" value, so both "Small rise" values would increase, i.e. "Small drop" would decrease and "Very large rise" would increase even further without any drops increasing at the same time.

Right stabilization level suggests, that initial values are very close to right ones:



But then... Guess what? No matter, how hard I tried - I wasn't able to do it. So what? PSR rating isn't only biased towards increasing. PSR system - is actually lie. PSR - is just an XP bar, where you can rise, but it's almost impossible to drop.



The only way to achieve stable rating with my stats - to almost DOUBLE rating change values for Win. So, I can say with all responsibility, that PSR is essentially an XP bar, cuz rating changes for victories have at least 2x multipliers applied to them.



Most probable PSR calculation parameters:



Conclusion: PSR - isn't just biased toward increasing. It's literally DOUBLE-biased towards increasing. So, it's just an XP bar and it's intention - to give players illusion of progression and make them overestimate their skill.

Edited by MrMadguy, 29 January 2018 - 12:20 AM.

[BTGbullseye]

The problem with your "how it should work" is the same as with the "how it works now"... It stays the exact same at each tier, instead of having the stabilization level increase.

200MS for T5, 250 for T4, 300 for T3, 350 for T4, and 400 for T1 would be a smarter set of stability areas. (or reduce each by 50 if you don't think having a 400 MS average to be a good T1 stability level)

Edited by BTGbullseye, 15 January 2018 - 12:40 AM.

[MrMadguy]

BTGbullseye, on 15 January 2018 - 12:37 AM, said:

The problem with your "how it should work" is the same as with the "how it works now"... It stays the exact same at each tier, instead of having the stabilization level increase.

200MS for T5, 250 for T4, 300 for T3, 350 for T4, and 400 for T1 would be a smarter set of stability areas.

No, it shouldn't. Skill level - is only thing, that should change with Tier. Cuz 200MS for Tier 5 - is completely different beast, than same 200MS, but for Tier 1. Skill - is relative thing. Ability to earn the same match score, but against more skilled enemies - is what is actually measured by MM.

[BTGbullseye]

That only applies if you're getting matched with only your tier... You're not. Player population is way too low to do that. You're getting matched with up to 2 tiers above and below your own tier. If you as a T1 can't get consistent 350+MS (my edited suggestion) against T2&3 players, you shouldn't be in T1.

[MrMadguy]

BTGbullseye, on 15 January 2018 - 12:50 AM, said:

That only applies if you're getting matched with only your tier... You're not. Player population is way too low to do that. You're getting matched with up to 2 tiers above and below your own tier. If you as a T1 can't get consistent 350+MS (my edited suggestion) against T2&3 players, you shouldn't be in T1.

I guess, many people here confuse skill rating system with leaderboard system. They are actually not the same.

[MrMadguy]

UPD: The only way to achieve stable rating with my stats - to almost DOUBLE rating change values for Win. So, I can say with all responsibility, that PSR is essentially an XP bar, cuz rating changes for victories have at least 2x multipliers applied to them.

Edited by MrMadguy, 15 January 2018 - 01:27 AM.

[adamts01]

MrMadguy, on 15 January 2018 - 12:17 AM, said:

As you know, I have always had some suspicions, that our PSR MM is broken.

Yes, PSR and MM are broken. I also think it's important to make clear that PSR and MM aren't at all related. How players are ranked and how players are grouped are two separate issues, both in a bad situation. "How PSR should work" is broken as well. At a very quick glance you can see that PGI's Moderate Drop combined with a Small Drop averages a more sever small drop. But on your chart, a small drop and a small drop average a moderate drop?.... That's not how averages work. But let's forget all that.

Any system heavily based on Match Score is doomed to fail, as two teams each putting up a good fight, which should be the end goal, will see every player progress. We could still see a situation where everyone in the match gets a PSR increase, which is part of our current problem.

So here's the simple solution, which accounts for support builds doing a good job, team leading, awareness, calling good targets...

• PSR is purely based on W/L alone
  
• We have 3 tiers, and someone has to bump someone down from tier 1 to move up from tier 2
  
• You only fight your own tier
  
• Each tier composes 1/3 of the active playerbase
  
• You lose 1 tier per month of inactivity
  
• Your first 50 matches back after a month gone yield double points

[The Basilisk]

As long as the developers try to use player statistiks based ratings for matchmaking only it can not function to any usefull extend.

Sure it is the player that makes the mech work but on the other side .... it is irreleant how well the best player can do headshots with a waterpistol against the biggest noob in MWO.

As long as matchmaking does not take into account the used chassis and equipment the whole thing a moot point not worth the discussion.

[MrMadguy]

adamts01, on 15 January 2018 - 01:40 AM, said:

Yes, PSR and MM are broken. I also think it's important to make clear that PSR and MM aren't at all related. How players are ranked and how players are grouped are two separate issues, both in a bad situation. "How PSR should work" is broken as well. At a very quick glance you can see that PGI's Moderate Drop combined with a Small Drop averages a more sever small drop. But on your chart, a small drop and a small drop average a moderate drop?.... That's not how averages work. But let's forget all that.

Any system heavily based on Match Score is doomed to fail, as two teams each putting up a good fight, which should be the end goal, will see every player progress. We could still see a situation where everyone in the match gets a PSR increase, which is part of our current problem.

So here's the simple solution, which accounts for support builds doing a good job, team leading, awareness, calling good targets...

• PSR is purely based on W/L alone
  
• We have 3 tiers, and someone has to bump someone down from tier 1 to move up from tier 2
  
• You only fight your own tier
  
• Each tier composes 1/3 of the active playerbase
  
• You lose 1 tier per month of inactivity
  
• Your first 50 matches back after a month gone yield double points

This horse is beaten till death. Purely W/L based MM isn't possible in team-based game, as teams are formed according to PERSONAL rating, but at the end of match your PERSONAL rating is adjusted according to TEAM PERFORMANCE. And smaller your contribution towards victory in comparison to other players is - less accurate such MM is. 1/12 towards victory and 1/24 of total contribution into result of match - isn't accurate enough to use purely W/L based MM, sorry.

Edited by MrMadguy, 15 January 2018 - 02:35 AM.

[MrMadguy]

Most probable PSR calculation parameters:

[adamts01]

MrMadguy, on 15 January 2018 - 02:14 AM, said:

This horse is beaten till death. Purely W/L based MM isn't possible in team-based game.

Even 100 matches and all that randomness is enough time to show that individual performance is the most powerful variable. Every other variable considered by PSR is just too hard to farm or isn't represented, just as drop calling isn't represented but crucial. In the end, better players will win more matches, period. You'd be hard pressed to find a decent player who disagrees that W/L isn't directly tied to skill after a decent sample size. It seems to me that the only players who don't put weight in W/L are those who just can't accept responsibility for their teams, like people who didn't get to the fight till it was over because their Atlas goes 46, or a LRM 80 Crab who's team didn't give them locks. Bringing/building a competent mech is part of W/L, same as skill.

[BTGbullseye]

adamts01, on 15 January 2018 - 03:25 AM, said:

better players will win more matches, period.

No, better teams will win more matches, period. You could be an awesome player, but if you get paired with a POS team, you will still lose to a mediocre team, because 1 awesome player can't carry an entire team of idiots. That's the problem with a purely W/L rating system, a single player rarely makes the difference between win or loss.

You shouldn't be saddled with responsibility for your team if nobody on your team is directly accountable to you.

[adamts01]

BTGbullseye, on 15 January 2018 - 03:31 AM, said:

No, better teams will win more matches, period. You could be an awesome player, but if you get paired with a POS team, you will still lose to a mediocre team, because 1 awesome player can't carry an entire team of idiots. That's the problem with a purely W/L rating system, a single player rarely makes the difference between win or loss.

You shouldn't be saddled with responsibility for your team if nobody on your team is directly accountable to you.

A single player often does make the difference between winning or losing in a 12v12. You can pretty much call tier 1 matches when a certain few guys show up. But that aside, a better player will absolutely win more matches than a worse player, given a large enough sample size. Sample size is the key here. I think 100 matches is plenty to negate the effect of bad teammates.

And look, if there was a better way to do this then I'd be all for it, but W/L is simply the most telling stat, given enough matches played. PGI has tried countless different stats but they can't account for some crucial contributions and everything they use can be farmed. So I say skip to the end result and measure W/L more heavily or on its own.

Edited by adamts01, 15 January 2018 - 03:50 AM.

[MrMadguy]

adamts01, on 15 January 2018 - 03:47 AM, said:

A single player often does make the difference between winning or losing in a 12v12. You can pretty much call tier 1 matches when a certain few guys show up. But that aside, a better player will absolutely win more matches than a worse player, given a large enough sample size. Sample size is the key here. I think 100 matches is plenty to negate the effect of bad teammates.

That happens only when Tier 1 guy plays on Tier 5 alt account.

[justcallme A S H]

BTGbullseye, on 15 January 2018 - 03:31 AM, said:

No, better teams will win more matches, period. You could be an awesome player, but if you get paired with a POS team, you will still lose to a mediocre team, because 1 awesome player can't carry an entire team of idiots.

Absolutely you can.

I do it plenty and I see others do it regularly as well. There are maybe 100 players out of 28,000 that play (regularly) right now that I'd be genuinely concerned about in QP. The rest, it's meh. I know I can beat them so as long as I don't derp, more often than not, I know it'll be a win in my teams favour because I can/do carry plenty. Granted that is the exception to the rule.

Edited by justcallme A S H, 15 January 2018 - 04:08 AM.

[MrMadguy]

justcallme A S H, on 15 January 2018 - 04:06 AM, said:

Absolutely you can.

I do it plenty and I see others do it regularly as well. There are maybe 100 players out of 28,000 that play (regularly) right now that I'd be genuinely concerned about in QP. The rest, it's meh. I know I can beat them so as long as I don't derp, more often than not, I know it'll be a win in my teams favour because I can/do carry plenty. Granted that is the exception to the rule.

Again, this happens just because you're being matched with/against lower Tier players, so you really can win matches solely it this case. But you should know, that it happens just because current MM is broken and you should also know, that for lower Tier players situation is completely opposite. For example I can go AFK and it won't change result of match, cuz my <100 dmg and 0 kills aren't such big deal.

Edited by MrMadguy, 15 January 2018 - 04:17 AM.

[arcana75]

I think a fair and balanced MM requires more understanding of what's going on and how to measure it than PGI is capable of. Based on this, there's actually no more real reason to pontificate on this except to pass time.

[adamts01]

arcana75, on 15 January 2018 - 04:22 AM, said:

I think a fair and balanced MM requires more understanding of what's going on and how to measure it than PGI is capable of. Based on this, there's actually no more real reason to pontificate on this except to pass time.

I don't think anything short of human-level AI can accurately determine a players actual contribution, be it damage, kills, leadership or moral. And I agree that this is beyond PGI. This is why I'll always argue for W/L, it skips all the meaningless stats and gets to the end result of how good a player is, on average.

MrMadguy, on 15 January 2018 - 04:16 AM, said:

Again, this happens just because you're being matched with/against lower Tier players, so you really can win matches solely it this case. But you should know, that it happens just because current MM is broken and you should also know, that for lower Tier players situation is completely opposite. For example I can go AFK and it won't change result of match, cuz my <100 dmg and 0 kills aren't such big deal.

That's a valid point about skill being tied to ability to influence the match. But you really need to stop blaming MM. We might have the best MM in the world, yet it's handicapped by an obviously broken PSR. Another thing to consider is that we just don't have enough players to support 5 tiers. No matter which ranking system you like, yours or my W/L proposal, consider 3 tiers equally populated by the game's players and a tier 2 having to push a tier 1 player out of his position. I see that as the best way to rank players, not by some number, but against the community.

[arcana75]

adamts01, on 15 January 2018 - 04:34 AM, said:

I don't think anything short of human-level AI can accurately determine a players actual contribution, be it damage, kills, leadership or moral. And I agree that this is beyond PGI. This is why I'll always argue for W/L, it skips all the meaningless stats and gets to the end result of how good a player is, on average.

And I would argue for a dynamic system based on monthly average match score. But we both won't get what we want, will we?

[adamts01]

arcana75, on 15 January 2018 - 04:51 AM, said:

And I would argue for a dynamic system based on monthly average match score. But we both won't get what we want, will we?

PGI sometimes listens to players. We eventually got CT armor on all Jenners and LMGs got a crit reduction. It's a possibility. I just think Match Score isn't all that reliable of an indicator. I've sat behind cover in my Kit Fox and shot down hundreds of missiles that would otherwise have hit a wall. Combined with mechs happening to step in to ECM coverage when the missiles wouldn't have hit anyway I made out like a bandit. That plus "scouting" rewards... W/L is the way to go. But either way, 3 tiers and the necessity to bump someone else down if you want to move up would solve this XP bar nonsense.