Jump to content

Psr Update And Hold On Patch.


713 replies to this topic

#441 David Sumner

    Member

  • PipPipPipPipPipPip
  • FP Veteran - Beta 1
  • 470 posts
  • LocationAuckland, New Zealand

Posted 11 June 2020 - 10:58 PM

View PostNightbird, on 11 June 2020 - 03:54 PM, said:


Another point on how easy it is to do rolling average:

Today we have a saved variable per player called PSR, and after a match we have a PSR_change. So the code for the new PSR is:
new_PSR = old_PSR + PSR_change.

To do a rolling average for MS over the last 50 games, we only need to save two variables, avgMS and N (N is matches played). After a match we have currentMS score.The code for the new avgMS is:

if N<50 then new_avgMS = old_avgMS * N/(N+1) + currentMS / (N+1)
if N>=50 then new_avgMS = old_avgMS * 49/50 + currentMS / 50
new_N = old_N+1

To do a rolling average for W/L over 50 games, likewise we need to save just two variables WLR and N. After a match we get a RESULT which is 1 for a win or 0 for a loss. Code:

if N<50 then new_WLR = {N * [old_WLR/(old_WLR + 1)] + RESULT} / {N * [1 - old_WLR/(old_WLR + 1)] - RESULT + 1}
if N>=50 then new_WLR = {49 * [old_WLR/(old_WLR + 1)] + RESULT} / {49 * [1 - old_WLR/(old_WLR + 1)] - RESULT + 1}


Takes a programmer 1 hr to do all of this. A new MM would take time but just a rolling MS or WL average took me 10 minutes to figure out.


Not even that.

Just assume N=100 always.

When you first start (a player, or weight class, or a chassis, or a variant), seed it with N dummy match scores.

New player = "N * target to be average" (or 90% of that if we assume they suck to start)
New weight class = "N * player average" (or 90% of that if we assume they suck to start) - means your first mech assumes you are 81% of average.

New chassis = "N * weight class average" (or 90% of that if we assume they suck to start) - means your first mech assumes you are 72% of average.

New variant = "N * chassis average" (or 90% of that if we assume they suck to start) - means your first mech assumes you are 63% of average.

(Or you just seed player, and mech 1 at 90% straight up)

Weight Class PSR is calculable as SUM(PSR for class variants) / number of variants on the rare occasion you need it,
Chassis PSR is calculable as SUM(PSR for chassis variants) / number of variants when you need it,


Let's say your "rolling average" PSR for your BLR 1G = 250 * 100 = 25,000. And that's your "player" PSR as well.
This match you get 280. 25,000 - 250 + 280 = 25,030. 25,000 - 250 + 280 = 25,030.
There's your new PSR with 100 match rolling average.

You drop in your first match in your new BLR 1D. You get given a PSR of 25,030 * 0.9 = 22,527
This match you get 250. 22,527 - 225 + 250 = 22,552. 25,000 - 250 + 250 = 25,000.
Next match you get 240. 22,552 - 226 + 240 = 22,566. 25,000 - 250 + 240 = 24,990.

Then let's say once a month, you go
"The average PSR is 20,000"
Player PSR = Player PSR + ((20,000 - Player PSR) * 0.05)

e.g. 25,000 + (0.05 * -5,000) = 25,000 - 250 = 24,750

BLR-1G PSR = BLR-1G PSR + ((20,000 - BLR-1G PSR ) * 0.05)
BLR-1D PSR = BLR-1D PSR + ((20,000 - BLR-1D PSR ) * 0.05)
...


And you use "PSR" instead of W/L because you can change PSR more subtly than W/L.
Even make it = W/L.

Edited by David Sumner, 11 June 2020 - 11:22 PM.


#442 David Sumner

    Member

  • PipPipPipPipPipPip
  • FP Veteran - Beta 1
  • 470 posts
  • LocationAuckland, New Zealand

Posted 11 June 2020 - 11:16 PM

View Postyrrot, on 11 June 2020 - 09:21 PM, said:

snip


Literally this part. This isn't a hard change, but it's now touching 1) the end of match logic, 2) the queries that run end of game, 3) the player database, 4) the matchmaker as a go between to hand off the new values you need in the game server to computer change in PSR. So now a change that was going to change float values in a table on the game server would be touching 2 game critical systems that require programmer intervention rather than a designer changing values. Plus QA scope creep.


It depends on how they've segregated the systems though.

I'd assume that "enough" data has to go from the end of game screen to the database to capture the stats data you can see on your profile: W/L, K/D, C bills, XP, Map performance, mech performance, Weapon performance.

Which means that, changing none of that, you should be able to do a "PSR" based on mech, map, player, W/L, and (I presume) match score though that doesn't show in your stats.

Being in a group might be the biggest problem for tracking a PSR specifically associated with it.

If they do ANYTHING other than change the relative weightings of the triggers, #1 is going to have to change, and #2 might. But #1 and #2 might be the same thing/operation.

#3 would probably have to change regardless, but might not. But I'd REALLY like PSR by variant, if not variant + map.

#4 might or might not change, depending on whether we are going "just do a zero-sum PSR to start" or not. That's a completely separate issue. If the MM currently only needs "these are the groups" and "these are the player PSR scores" then you could potentially work it by manipulating the PSR scores you pass it.

Edited by David Sumner, 11 June 2020 - 11:16 PM.


#443 Dauntless Blint

    Member

  • PipPipPipPipPipPip
  • Fire
  • Fire
  • 408 posts
  • LocationPlaying other games.

Posted 11 June 2020 - 11:19 PM

View PostFlynnTheAvatar, on 11 June 2020 - 10:23 PM, said:


Great idea, I also think the MM should take the chassis in account.

But I would not modify the amount of PSR gained/lost based on the chassis. I would say that MM modifies your effective Tier based on the chassis. So, if you use an unskilled and/or lemon chassis, MM should treat you as a Tier 2 or even Tier 3 player, even if you are normally Tier 1. And perhaps put you in a higher Tier if you run an OP build.

So:
- unskilled mech: effective Tier + 1
- known bad chassis: effective Tier + 1
- OP chassis: effective Tier - 1

The reason is: If you really skill a new chassis, or you are running around in a troll / fun build you are not playing at your best. Playing in a lower tier might result in more enjoyable games where you might make a difference instead of just being carried.


The opposite. Strong performance chassis push you up +1 (harder games). Poor performing chassis takes you down -1(non meta, grinding skillpoints)

Remember the goal of the PSR is to try and make a match that is a perfect draw with MM.
(too many people think it's about vanity first) I'm not saying you either of course.

It should be: Player (W/L x MS) + Chassis (W/L x Chassis Global average MS) = Effective match PSR.
True PSR: is Player W/L x MS (or how you go up and down the ranks could add K/D to that)

It has the effect of taking serious builds up and skillgrind/nonmeta away from the serious guys.
Solo que is the sandpit of MWO.
Some elitists are annoyed if your not bringing a meta build and this would help separate them from each other.

How the global chassis average updates could be weekly or monthly if not live- ~not an engineer.

Edited by Dauntless Blint, 11 June 2020 - 11:39 PM.


#444 FlynnTheAvatar

    Member

  • PipPip
  • The Tip of the Spear
  • The Tip of the Spear
  • 28 posts
  • LocationBad Homburg, Germany

Posted 11 June 2020 - 11:48 PM

View PostDauntless Blint, on 11 June 2020 - 11:19 PM, said:

The opposite. Strong performance chassis push you up +1 (harder games). Poor performing chassis takes you down -1(non meta, grinding skillpoints)


We mean the same thing. I was referring to the actual tier name. So, if you are normally Tier 2, and you run a strong build, you should be pushed to Tier 1. Bad chassis should take you to Tier 3.

#445 Dauntless Blint

    Member

  • PipPipPipPipPipPip
  • Fire
  • Fire
  • 408 posts
  • LocationPlaying other games.

Posted 12 June 2020 - 12:08 AM

Player(W/L x MS x K/D) + Chassis(W/L x chassis Global average x Chassis average K/D) = Effective match PSR

Player(W/L x MS x K/D) = True PSR.

#446 Gagis

    Member

  • PipPipPipPipPipPipPipPip
  • FP Veteran - Beta 1
  • FP Veteran - Beta 1
  • 1,731 posts

Posted 12 June 2020 - 12:52 AM

Chassis, weapon and class performance is such nuance that we don't really need to track that. The diffrence between how a player performs in their best mech and their worst mech is smaller than the typical difference between skill level of two players. Tracking that would be unnecessarily complex for little to no gain.

#447 Brizna

    Member

  • PipPipPipPipPipPipPipPip
  • Liquid Metal
  • Liquid Metal
  • 1,363 posts
  • LocationCatalonia

Posted 12 June 2020 - 01:00 AM

A commission to study the issue is and old time proved way of getting nothing done, the more people you involve in the commission the more chances it just wastes time.

JUST GET IT DONE.

A PSR RESET IS A GOOD THING

DON'T DELAY IT STUDYING IT



#448 RockmachinE

    Member

  • PipPipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 1,155 posts

Posted 12 June 2020 - 06:21 AM

The sad thing in this debate is that with a dwindling playerbase people of different skill levels will be forced to play together either way. This attempt at making a fair PSR rating and thus tiers that are reflective of skill will more or less be in vain when we have such a small player pool to draw from in the first place.

I don't believe it will have a major impact on the quality and longevity of the game. Though I am happy that PGI is showing active interest in MWO.

#449 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 12 June 2020 - 06:22 AM

View PostMadcatX, on 11 June 2020 - 10:08 PM, said:

Nightbird, I have a few questions or require clarifications about the "hidden skill" element of your data analysis, just to make sure that I, and anyone else, know exactly what that refers to.

1) Are a person's WLR in both the current and proposed MM the only factor that would contribute to the hidden skill?

2) When you refer to it as being a "hidden" value in current MM, is it hidden because current MM is matching based mostly off of Tier ranking. Thus Tier 1 players having a hidden value that, if the tier system was actually representative of skill and not mostly an XP bar, a lot of Tier 1 players would be in a lower tier currently?

3) If I understand the new MM proposal you have made, it would make the entire Tier system obsolete would it not.

4) An interesting point was made a few posts ago regarding high-performance chassis (I assume he's referring to how well it performs overall or using a meta build on it as well). I too bounce often between running meta builds and non-meta builds. If I'm being matched solely off of WLR, the MM can't tell what I'm running. Is you're mindset (and perhaps data you've analysed) that my WLR would basically average out my skill level and use that for MM purposes?

Anywho, keep up the good work.

- Not a programmer.


1.

The concept of hidden_skill is that it represents the unknowable true skill each person has, a value that completely and perfectly represents the average player contribution to a team. It is unknowable because we can't see this value, so we can only try to predict it using that person's performance metrics, things like avgMS or WLR. The thread in my second link tests which of avgMS, WLR, and other collected metrics are the best at predicting true skill, and WLR wins so that's why I used WLR in the proposal. (The current MM tries to predict true skill using the XP bar)

In the simulation, the Match Maker has to create the two teams without the use of the hidden_skill value. It can only use predictors like WLR. After the two teams are assembled by the MM, the match simulator uses the true hidden_skill to determine whether the MM did a good job or not by calculating the team 1 win chance. The closer to 50% the true win chance is the better the job the MM did, the further away the worse. The match simulator also uses this win chance to randomly determine the winner and whether the match was a stomp, and uses the win or loss to update the WLR of all participants. This process is repeated 10,000 times.

2. Hidden_skill is the true skill of each player. It is hidden because the Match Maker can't see it and use it to make matches, it can only use player stats we have to predict it and better predictions lead to better matches on average. I said Tier 1 just to avoid implementing Tiers in the simulation, I probably should have said I discarded Tiers completely. In the simulation, Tiers or PSR = WLR. If the Tier system was not an XP bar, then a lot of people in Tier 1 would drop out, yes.

Worth mentioning the skill band I created is wide. 200 to 1800 is just numbers. To put it in perspective, if team A had 12 hidden_skill 1000 (average) players, and team B has 11 skill 1000 players and 1 skill 1800 player, team B had a true win chance of 84%. This large influence a skilled player can have on an otherwise pug vs pug fight I think matches what we see in game.

3. I'm saying we can keep the XP bar as is, since we don't want to show WLR on the front to the player. As long as it's not messing with the MM, then an XP bar gives people a sense of accomplishment and that's good. If anything, we should make it so that it's impossible to lose XP bar, so that people will stop doing things in games to earn MS instead to helping the team win. If the OP idea is implemented, EVERYONE will be grinding to avoid losing the XP bar LOL. LRMS and AMS everywhere.

4. Even for my proposal I'm asking for more changes than what PGI is willing to do, and chassis specific PSR is a much larger ask (I would prefer it if possible but it's unfortunately not). The proposed rolling WLR at least will have an improvement over today. After switching high to low performance chassis, you will lose more for a few games until the system predicts your contribution correctly. This goes both ways, when you switch from low to high performance chassis, the system will underestimate you and you'll win more, so it evens out over time. In both cases, the correctional period is temporary. Today you should only use high performance chassis because the system doesn't adapt to anything and thus using meta mechs gives you a permanent advantage.

Edited by Nightbird, 12 June 2020 - 08:40 AM.


#450 OneTeamPlayer

    Member

  • PipPipPipPipPipPip
  • Ace Of Spades
  • 399 posts

Posted 12 June 2020 - 07:14 AM

How are we intended to come to a community consensus from a 20+ page thread which over the course of a week has had zero developer input on any of the suggestions?

You said matchmaker is making 1 in 3 games a stomp and we're on 3 days past when the fix for that was intended to be implemented with no visible progress whatsoever and so many disparate views that it seems quite apparent that the "community" is fractured in multiple different ways as to a positive outcome.

When is PGI going to step in, pick an option, and move forward?

Every one of us here knows that its impressive to get 10% of players to read a thread, even one highlighted in launcher cause most don't pay attention to that in favor of getting to launching games ASAP.

Of that 10% how many do you think are still reading past page 5. How about page 10? How about day 3 and page 22?

This thread has outlived its functionality and needs to be closed, compiled into a list of suggestions for voting, and then posted as a poll with only the explanations for systems and zero discussion (with a separate thread for discussion of the choices but not more suggestions).

#451 Gagis

    Member

  • PipPipPipPipPipPipPipPip
  • FP Veteran - Beta 1
  • FP Veteran - Beta 1
  • 1,731 posts

Posted 12 June 2020 - 07:28 AM

There's a number of in-depth community-designed proposals including a better version of mine in the works. To be released today or tomorrow.

#452 FearThePaladin

    Member

  • PipPipPipPipPip
  • Guillotine
  • Guillotine
  • 117 posts

Posted 12 June 2020 - 07:29 AM

Make sure you update the UI to reflect how much you moved up or down.

#453 Brauer

    Member

  • PipPipPipPipPipPipPipPip
  • 1,066 posts

Posted 12 June 2020 - 07:57 AM

View PostDavid Sumner, on 11 June 2020 - 10:32 PM, said:


Only because in the game, no one cares about "living to take part in the next mission".
There's basically no repair cost, no rebuild time, no injury, no resupply limits.

It would be a d**n sight different if you sent 12 people in one end of a 3 mission run, each mission facing 4-8 opponents, and a limited resupply capacity until they came out the other end.


Repair and rearm will only punish players who are bad at the game by increasing the grind for them or making them perpetually cbill poor. It doesn't do anything to change the fact that the best way to win is still to take your opponents off the board.

Also, no idea why you are proposing mismatched team sizes with 4-8 people having to take on 12 for three missions in a row. You realize this is a PVP game right and that you need people to sign up to play on both teams in a match?

View PostDavid Sumner, on 11 June 2020 - 10:37 PM, said:

I've NEVER seen that.


Have you ever looked at the breakdown? Typically heavies and assaults make up probably 60-80 percent of of all mechs dropping at any given time. Lights often make up 10-15% of all mechs dropping.

#454 Z Paradox

    Member

  • PipPipPipPipPip
  • Bad Company
  • Bad Company
  • 102 posts
  • Locationozz

Posted 12 June 2020 - 08:35 AM

View PostOneTeamPlayer, on 12 June 2020 - 07:14 AM, said:

How are we intended to come to a community consensus from a 20+ page thread which over the course of a week has had zero developer input on any of the suggestions?



it would be nice to hear from devs what is and what isn't possible to change...

#455 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 12 June 2020 - 10:40 AM

View PostZ Paradox, on 12 June 2020 - 08:35 AM, said:

it would be nice to hear from devs what is and what isn't possible to change...


Part of the problem is that there is no point to changing any of the things devs say can change. The only thing on the table is changing formula to Match Score and how it impacts PSR. However:

No matter how you weigh the components going into match score calculations, every player will have an average PSR change per game. Let's call this X.

After 100 games, your "skill" = 100X. After 1000 games, your "skill" = 1000X. After 5000 games, your "skill" = 5000X.

Is your true skill actually increasing by a factor of 10, 50, 100 over time? Or is this in reality not a measurement of skill? It's not. It's an EXP bar that is disjointed from any representation of your skill. If you have a PSR=5000, the MM doesn't know if you earned it with 5 * 1000matches, or with 50*100matches. The latter would obviously be a much better pilot than the former.


My suggestion: https://mwomercs.com...32#entry6337632

See the first link for a MM proven to work.
See the second link for proof MS is useless.

Edited by Nightbird, 12 June 2020 - 10:47 AM.


#456 The Teddy Bear

    Member

  • PipPipPip
  • Bad Company
  • Bad Company
  • 87 posts
  • LocationSomewhere cuddling

Posted 12 June 2020 - 11:08 AM

View PostNightbird, on 12 June 2020 - 10:40 AM, said:


Part of the problem is that there is no point to changing any of the things devs say can change. The only thing on the table is changing formula to Match Score and how it impacts PSR. However:

No matter how you weigh the components going into match score calculations, every player will have an average PSR change per game. Let's call this X.

After 100 games, your "skill" = 100X. After 1000 games, your "skill" = 1000X. After 5000 games, your "skill" = 5000X.

Is your true skill actually increasing by a factor of 10, 50, 100 over time? Or is this in reality not a measurement of skill? It's not. It's an EXP bar that is disjointed from any representation of your skill. If you have a PSR=5000, the MM doesn't know if earned it by 5 * 1000matches, or with 50*100matches. The latter is obviously an incompatibly better pilot.


My suggestion: https://mwomercs.com...32#entry6337632

See the first link for a MM proven to work.
See the second link for proof MS is useless.


As I said earlier, I think W/L will be a tough one after the arrival of groups in the que. There should be no doubts any longer that a high end group player, if having similar skill in the group, will easily get themselves into somewhere along 4-10 W/L while a high end solo player will struggle badly to get to 2 W/L in this que.

I think that after some months with W/L online you will indeed have created a soup. Think about it, people playing solo/group 10/90, 20/80....90/10 etc - their W/L will be all over the place, even if they are on the same skill level. At any time someone changes from their main playing pattern, they will direcly imbalance the que, when slotted in according to you original simulation.

However, I think slotting in players after MS instead of W/L in your simulation would have produced a perfectly acceptable outcome as well, even if not as perfect as W/L. The slotting of the players are the golden nugget in your proposal.

I think MS will be a more stable parameter than W/L when you switch between groups or just playing solo in the new que. The main challenge to your proposal is, how do you slot in the solos if you have random groups to play with, or did you suggest anything about balancing groups that I missed?

It would be very interesting to see your simulation with random groups while still slotting in the solo in line with your original proposal to see what happens then.

Edited by The Teddy Bear, 12 June 2020 - 11:35 AM.


#457 Dauntless Blint

    Member

  • PipPipPipPipPipPip
  • Fire
  • Fire
  • 408 posts
  • LocationPlaying other games.

Posted 12 June 2020 - 11:32 AM

View PostGagis, on 12 June 2020 - 12:52 AM, said:

Chassis, weapon and class performance is such nuance that we don't really need to track that. The diffrence between how a player performs in their best mech and their worst mech is smaller than the typical difference between skill level of two players. Tracking that would be unnecessarily complex for little to no gain.


I disagree, It's the difference between 4-6 kills in a match and zero kills for 200 damage. It's far bigger than the player.

And it's not complex to do the math. A chassis change can become 50%(for arguments sake) of the entire teams power. A chassis can be worth 4-6 players by the end of the match.

I also don't understand why people think it has to be only W/L or only MS they should augment each other.

Like wise player + chassis average( W/L x MS x K/D ) to infinity. The only question is do they want to spend a buck on the engineer. Meta chassis definitely have a better K/D ratio regardless of the player.
Because this is the test bed for the overhaul or reboot to get it right.

Edited by Dauntless Blint, 12 June 2020 - 12:18 PM.


#458 MrMadguy

    Member

  • PipPipPipPipPipPipPipPipPip
  • 2,225 posts

Posted 12 June 2020 - 11:34 AM

Worst case scenarios:
1) This is done only to actually delay this change
2) This is done only to bait us, i.e. to increase our interest towards game, while not doing anything useful
3) This is done only to remove any responsibility from devs, i.e. say "it's players choice - not ours"

#459 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 7,518 posts

Posted 12 June 2020 - 12:09 PM

View PostThe Teddy Bear, on 12 June 2020 - 11:08 AM, said:


As I said earlier, I think W/L will be a tough one after the arrival of groups in the que. There should be no doubts any longer that a high end group player, if having similar skill in the group, will easily get themselves into somewhere along 4-10 W/L while a high end solo player will struggle badly to get to 2 W/L in this que.

I think that after some months with W/L online you will indeed have created a soup. Think about it, people playing solo/group 10/90, 20/80....90/10 etc - their W/L will be all over the place, even if they are on the same skill level. At any time someone changes from their main playing pattern, they will direcly imbalance the que, when slotted in according to you original simulation.

However, I think slotting in players after MS instead of W/L in your simulation would have produced a perfectly acceptable outcome as well, even if not as perfect as W/L. The slotting of the players are the golden nugget in your proposal.

I think MS will be a more stable parameter than W/L when you switch between groups or just playing solo in the new que. The main challenge to your proposal is, how do you slot in the solos if you have random groups to play with, or did you suggest anything about balancing groups that I missed?

It would be very interesting to see your simulation with random groups while still slotting in the solo in line with your original proposal to see what happens then.


First of all, you're talking about average Match Score right? That is also not one of the accepted options in this thread. PSR by MS in the OP will never stabilize at any value other than artificial caps.

The reason that average MS is bad is that it doesn't recognize groups at all when creating teams. Playing in a group is a major advantage to a player. You want a MM that recognizes that a grouped player is more dangerous than a solo player. By saying avgMS is stable, you're basically saying it cannot recognize a player becoming more effective and dangerous.

WL seesawing is good because the contribution of the player to the team is seesawing as he changes between playing solo and playing in a group. WL reflects that change, avgMS doesn't.

Edited by Nightbird, 12 June 2020 - 12:14 PM.


#460 The Teddy Bear

    Member

  • PipPipPip
  • Bad Company
  • Bad Company
  • 87 posts
  • LocationSomewhere cuddling

Posted 12 June 2020 - 12:20 PM

View PostNightbird, on 12 June 2020 - 12:09 PM, said:


First of all, you're talking about average Match Score right? That is also not one of the accepted options in this thread. PSR by MS in the OP will never stabilize at any value other than artificial caps.

The reason that average MS is bad is that it doesn't recognize groups at all when creating teams. Playing in a group is a major advantage to a player. You want a MM that recognizes that a grouped player is more dangerous than a solo player. Only WL does that. By saying avgMS is stable, you're basically staying it cannot tell the difference between the two.

WL seesawing is good because the contribution of the player to the team is seesawing as he changes between playing solo and playing in a group. WL reflects that change, avgMS doesn't.


Well, the variance between solo and group W/L, compared to the variance in solo and group MS? Did you ever investigate that?





5 user(s) are reading this topic

0 members, 5 guests, 0 anonymous users