Psr Community Feedback - Round 1
#1
Posted 16 June 2020 - 04:30 PM
First off, thank you to everyone participating in this. There are some ideas here that will work with some fine tuning and we can work on those to get something that works for the game and you the community.
After reading through all the posts made so far, there seems to be a two sided trend going on. On one hand there's the notion of comparing an individual against the 24 other players in the game and separate of the win/loss condition. On the other hand, there's the notion of keeping the win/loss condition with varying methods of comparing the individual against their team or the players in the match. Most of the suggestions are permutations of these two systems.
For clarification let's make sure we use common terminology. For the rest of this write-up, I'll be using the following:
Match Score Kicker (MS kicker) - Variables that include things like Kills, Damage, Captures, Spotting, etc. that
Match Score (MS) - The resulting score based off Match Score Kicker formula.
Player Skill Rating (PSR) - A player's skill rating in a range from 0 to 3750 with new players starting at 1500. (This is essentially the players Elo rating)
PSR Movement - How much a player moves up or down in PSR based on their Match Score performance.
Match Maker (MM) - A server side algorithm that matches players into teams and games.
Let's take a look at the aspects of these two main suggested systems.
Core 1A: Global Compare
This is the system that compares the individual player against all other players in the match (23 others). Essentially this system compares a player's Match Score against all other player's and assigns PSR movement based on that comparison. This system works well as it can be made zero sum very easily. It also doesn't measure the player against an arbitrary threshold as all players will fall into discreet brackets in the resulted sorted list. This proposal also removes the win/loss component to PSR movement. The variations being suggested can be summarized in the image below:
All players in a match are ranked from highest to lowest Match Score independent of the team they were on. In the example image above, as long as the number of green squares matches the number of red squares, the number of yellow squares match the number of orange squares, and there is an even number of grey squares, you have a zero sum PSR movement formula. The number of green squares etc can be adjusted before implementation with the key criteria of symmetry on both sides of neutral (grey squares).
However, there are pros and cons to this system depending on what the numbers are set to. There are scenarios that all suggestions should be taking into consideration. For example, how much would you want to reward a team that lost? In some of suggestions around this global compare system, even mid level participation could end up in positive gains from players on the losing team. High performing players could have a bad match and be severely punished for that.
Core 1B: Global Scalar
This is the system that compares the individual player against all other players in the match (23 others). The difference between the above is how the PSR movement is determined. Instead of falling into discreet brackets, a formula is used to determine how much a player should shift in PSR movement.
In the image below, you can see how this has a more dynamic method of distributing PSR movement.
This also breaks zero sum but keeps the performance variable as high value in the outcome. This system also suffers the same cons as outlined in the non-scalar version above.
Core 2A: Win/Loss Team Compare
In this system, win/loss still dictates direction of PSR movement. However, instead of an individual measure against Match Score, the player's match score is compared to the match score of all players on their team. For example, the winning team will have a neutral to positive movement in PSR. The losing team will have neutral to negative movement in PSR.
In the following image you can see how this scenario works out:
Again, to maintain zero sum, the distribution of points in both positive and negative directions need to be equal.
Core 2B: Win/Loss Team Scalar
Similar to the above, but without the determined movement values, this system will compare players against their team mates and scale the PSR movement based on a formula that uses Match Score as an input. Basically after the win/loss condition determines PSR movement direction, a formula is used on each player's Match Score to determine their PSR movement relative to the average of the team's Match Score. There is an issue with this, it is no longer zero sum as the number of players moving up by X are not matched by the number of players moving down by X. You can see this in the image below:
Now what does this all mean?
Well, for starters, when it comes to these two systems, both are being suggested in a manner that is something we can implement. It works with current metrics that we have available.
There are some suggestions out there that will not work when trying to access data that isn't loaded while players are still connected to the game servers. This type of change is something we cannot address at this time.
From what I've been reading, there's a slight majority of players who are favoring Core 1A or 1B above. This doesn't negate the large number of people favoring 2A or 2B above. However, it does indicate where our starting point should be. There's nothing stopping us from trying 1A or 1B above and letting it run for a few months to see how people are settling into their Tiers.
What else is there to discuss?
While there has been a little discussion over Match Score Kicker values, there hasn't been a lot of consensus for a change in any direction. As you folks have been discussing here, there are so many player controlled variables that data cannot predict. This is why the numbers required for the kickers were pulled from averaging large numbers of historical matches across large numbers of players. Some of you are using random number generators to provide data samples which is really cool to see.
One thing both raw data and random number generators will always have a hard time determining is what happens when a player starts playing in an area in the PSR levels that they should be? i.e. Those magnificent Match Scores from earlier on are going to drop as you climb the PSR ladder. There's only so many enemies and so much damage that can be dealt. This is why PSR/Elo etc form a bell curve across the player distribution.
Internally we've been discussing some of the values assigned to some critical Match Score kickers. Like mentioned by many of you, damage weighting is too high. Way too high? Not really.. but undeniably high none the less. We think it's necessary to correct this PRIOR to implementing any of the suggestions above. The reduction amount we're looking at is between 10-15%. We don't want to break the ability of higher tonnage 'Mechs to get good match scores, but we also don't want damage to bury lighter tonnage 'Mechs either (which it kind of is doing right now). We will leave all other kickers in place for now as we monitor what matters to you in your feedback on the issue.
Next plan of action:
Lets assume Core 1A or Core 1B are the current marching orders. We need to determine which of the two is going to be favored by as many of you out there as possible. (Again, 2A and 2B are not out the window.. just on pause for now so we can try getting something implemented and testing.)
We also need suggestions on Match Score kicker values. You can come up with ANY set of numbers you want to try out but use that last scalar multiplier in the list to bring numbers down to a manageable level. i.e. your numbers shouldn't be shooting people into the high 1000 MS levels. I mean, they could, but just adjust the final scalar to bring the final number into a range like you currently see in the game.
Thanks again for all the suggestions and discussion in the other thread. I hope to see it continue here as well.
-Paul
#2
Posted 16 June 2020 - 05:21 PM
The current MM creates a wide WLR range from <.5 to >2. The best and most important reason this has been bad for MWO and PGI is that player attrition rate is lowest when WLR=1. People quit when they win too little (too painful) and when they win too much (too easy).
Calculated from Jarl's data:
Do a little calculation on overall pop and you can see the true monthly attrition rate is ~3% (not everyone that leaves for a season leaves forever).
If a Match Maker consolidated the WLR for all players into a 0.8-1.3 WLR region from the beginning of MWO, if you take the relative improvement from the graph above this would reduce the player attrition rate to 2%.
Such a change would result in a player base 3x the size today, and also generated for PGI 32% more lifetime revenue from MWO.
The MM I proposed on the last thread (with tweaks) would have done this. There are even better MM ideas (not presented due to difficulty) that would push the WLR for all players into a 0.95-1.05 region, and drive monthly attrition rate down to 1.5% and lifetime revenue up by 56%. (Yes I know this underestimates the benefits of a MM because it fails to take into account the influx of new players. Under-promise and all)
Amazing what a little MM can do right? Here's to hoping for success in the next multiplayer title.
Edited by Nightbird, 24 June 2020 - 09:17 AM.
#3
Posted 16 June 2020 - 05:28 PM
IMO just change the PSR stuff and change the MS kicker for damage after some time later, like a week or so.
This way the PSR changes can be evaulated more clearly without being affected by MS Kicker changes for damage changes.
Edited by OZHomerOZ, 16 June 2020 - 05:29 PM.
#4
Posted 16 June 2020 - 05:40 PM
I'd like to focus on the matchscore discussion below.
Why is it important that PSR+matchscore reflects wins, kills and damage?
By analyzing the stats of high level players we can see there is a strong correlation between have kill & damage stats and having a high WLR. In theory we should just be able to use WLR as a representation of player skill and to an extend that's true. However there are various factors in this game that reduces the accuracy of WLR (groups as one example) and the reality is that all 3 key indicators have flaws and accuracy issues. However when combined they provide a much better picture of player skill.
Using data from the Jarl’s leaderboard I can demonstrate this. I Here is a link to all players on the leaderboards who have played more than 1k games and have a WLR between 1.90 and 1.99 (https://leaderboard....gets%0D%0AEmdee).
I’m actually familiar with most players on the list from comp and/or QP and that’s a reasonably solid group. However looking through their stats we can see that despite them all having similar WLRs, there is a fairly wide gap in their displayed abilities. Now look what happens when I take 12 players from that list ranked by;
WLR only (https://leaderboard....ARubberDarkDuck);
WLR + K/D (https://leaderboard....%0ATerminator20);
WLR + Matchscore (https://leaderboard....%0D%0Awhite0Fox);
The list using WLR still contains noticeable skill gaps but when we also use matchscore or K/D then the grouping becomes much better. If we put these teams against each other, the KD and matchscore groups would be quite closely matched but the WLR only one would be the least competitive. This also holds true when we perform the same test using matchscore. Again here is a group of players with 1k+ games and matchscore between 346-355(https://leaderboard....ANuclear+Weapon).
Now again lets rank them by; matchscore only (https://leaderboard....01%0D%0AArcayne);
matchscore + WLR(https://leaderboard....D%0Avidjahgames);
matchscore + K/D(https://leaderboard....nee%0D%0Amalz79).
In these comparisons it’s easy to see that using 2+ indicators improves the grouping quite a bit. It's not perfect by any means but making sure that all 3 values are represented within PSR and/or matchscore will help improve the efficiency and effectiveness of the system. Keep in mind this is not about WLR being better than matchscore or vice-versa. It's about why using more than 1 value will help improve the accuracy of matchmaking.
So what’s wrong with the current matchscore formula?
PGI has does not want to give out the exact values involved in the matchscore formula. However through testing and theory-crafting, we have an understanding of the formula that is about 95-99% correct. Currently matchscore is mostly influenced by damage, then secondary bonuses (AMS, protect light/medium, etc) and kills (assists, killing blow, KMD, solo kills, components destroyed). Winning and losing only has a very minor effect on the final matchscore. Some examples are below;
Game 1 (matchscore 222): SK 0; KMD 2; KB 0; assists 5; dmg 331; (11% from kills, 64% from damage, 19% from passive bonuses, 6% from winning)
Game 2 (matchscore 272): SK 0; KMD 0; KB 0; assists 5 damage 0; (7% from kills, 0% from damage, 88% from passive bonuses, 5% from wining)
Game 3 (matchscore 359): SK 1; KMD 1; KB 1; assists 6; dmg 565; (20% from kills, 67% from damage, 9% from passive bonuses, 4% from winning)
Game4 (matchscore 510): SK 5; KMD 5; KB 7; assists 4; dmg 692; (32% from kills. 58% from damage, 9% from passive bonuses, 3% from winning)
Game 5 (matchscore 910): SK 5; KMD 7; killing blows 7; dmg 1459; (19% from kills, 69% from damage, 11% from passive bonuses, 1% from winning)
As we can see matchscore does not give a balanced representation of the 3 key performance indicators identified earlier. Kill related bonuses are far too low in many cases while damage and even passives tend to be too high. Killing blows, kills most damage and solo kills are all valued far too low compared to how much influence they have on a win. Winning at the moment makes almost no difference to the final score (winning is +28MS, losing is +14MS, and a tie is 0MS).
Solution for if Core 2A/Core2B is chosen
This makes things a lot easier. As wins/losses contribute directly to whether players gain or lose PSR, it is not as important to have them represented heavily within the matchscore formula. Therefore the recommended changes would be;
Balance Kills + damage bonuses so they represent 70-90% of the final matchscore. Killing blows and kills most damage should receive significant buffs while damage bonuses need to be reduced. 1 KB + 1 KMD should equal roughly 150-250 damage. Solo kills should also receive a slight buff but should be kept below KBs and KMDs due to the other conditions already being triggered when a solo kill happens. If the value of solo kills is too high then it will increase hostility in regards to perceived “kill stealing” behaviour (this was a large issue during the old leaderboard events).
Passive bonuses should be kept primarily as c-bill/exp earners but in order to keep matchscore as an assessment of player skill, it’s important not to allow passive bonuses to affect it too much (no more than 10% for higher MS games).
If Core 1A/Core1B is chosen
This makes things a little bit more complicated. Now we probably want to make sure that the win+kills+damage bonuses add up to 75-90%. However as the win is just a true or false condition, rather than a stacking condition, it makes deciding the actual bonus for getting a win somewhat tricky. For example if we say a win should be worth +50MS, for a final matchscore of 200 that is a weight of 25%. However for a 600MS game it’s now only 8.3%. For high scoring players, the win would have much lower value than for an average level player.
To solve this we could just give wins a consistent percentage weight. We simply add up the other matchscore kicker bonuses then apply the win bonus as a percentage bonus at the end. However this would run the risk of causing the matchscores to become excessively large. Therefore it’s probably better to instead apply this value as a percentage reduction for a loss rather than adding for a win. For example let’s assume wins/losses weight is 25% and then apply this to the 5 example games I gave above.
Game 1: win = 222ms; loss = 167ms
Game 2: win = 272ms; loss = 204ms
Game 3: win = 359ms: loss = 269ms
Game 4: win = 510ms; loss = 383ms
Game 5: win = 910ms; loss = 583ms
Giving wins/losses a percentage weight ensures that regardless of the skill level of the player, avoiding a loss will always have a significant impact of their final matchscore and thus on their PSR gains/losses.
Hopefully Paul will provide us with some more data samples regarding matchscore. After discussing this more with members of the community I hope to make a make a 2nd post with a spreadsheet containing more definite matchscore balance change values.
Edited by Dogmeat1, 17 June 2020 - 05:39 AM.
#5
Posted 16 June 2020 - 05:54 PM
Paul Inouye, on 16 June 2020 - 04:30 PM, said:
Core 2B: Win/Loss Team Scalar
Similar to the above, but without the determined movement values, this system will compare players against their team mates and scale the PSR movement based on a formula that uses Match Score as an input. Basically after the win/loss condition determines PSR movement direction, a formula is used on each player's Match Score to determine their PSR movement relative to the average of the team's Match Score. There is an issue with this, it is no longer zero sum as the number of players moving up by X are not matched by the number of players moving down by X. You can see this in the image below:
It is easy to ensure 0 sum and gives you control over the effect on PSR Win/Loss has compared to MS.
The 2 control variables are:
X = Win/Loss Component (Average PSR Shift amount for each team)
Y = Matchscore Component (Variance of PSR shift from X)
Win/Loss variable:
W = +1 for winning pilots and -1 for losing pilots.
Match variables:
P = Individual Pilot Matchscore
A = Team Average Matchscore
Equation:
PSR Shift = W*X+Y*(P/A-1)
This is 0 sum, scaled within the team and takes win/loss into account. Doc and Sheet linked for more detail.
CLICK LINKS FOR EXPLANATION IN DETAIL!!!!!!!!!!!!!!!!!!!
https://docs.google....3LgMBQJ4hE/edit
https://docs.google....dit?usp=sharing
EDIT 2: Modified version of 2B called 2C is up which incorporates both Team and total Match AVG MS to be more fair.
https://docs.google....dit?usp=sharing
Edited by Jay Z, 17 June 2020 - 06:04 PM.
#6
Posted 16 June 2020 - 06:03 PM
#7
Posted 16 June 2020 - 07:29 PM
Edited by Anharn, 16 June 2020 - 07:29 PM.
#8
Posted 16 June 2020 - 07:44 PM
I vote for 1A
I hope other folks in this thread just simply vote. We get something in place and can adjust MS values later if an improvement in MM is not observed.
#9
Posted 16 June 2020 - 07:52 PM
Edited by Domenoth, 16 June 2020 - 07:53 PM.
#10
Posted 16 June 2020 - 08:07 PM
If I absolutely must choose between the awful first two options, I will begrudgingly go with 1A, as there is slightly more lipstick on that pig.
I want to see an actual show of hands on this, first, though. I don't believe that the first two options have a significant amount of support over the last two.
#11
Posted 16 June 2020 - 08:17 PM
I have some suggestions for the MS however I also agree that this is something that should be looked at once the dust settles a little to have a better picture of the effects.
MS Changes
ams_missile_destroyed - Gain (reduced by 25% but Gain C-bills by 25%)
damagedone - Gain (reduced by 10 - 15%)
New MS ideas
LOS_damagedone - Gain (make up the reduced 10-15% from damagedone reduction)
The idea behind LOS_damagedone is to give an incentive to those who use indirect fire to increase their scores while giving those who already LOS a similar MS gain to what they have now.
survivalbonus - gain (starts at 75% total hp remaining and scales up value as the amount of hp is reduced)
75% totalHP% = 1MS
1% totalHP% = 75MS
This would give those who survive a bonus based on how much damage they took in the process.
#12
Posted 16 June 2020 - 08:27 PM
#13
Posted 16 June 2020 - 08:51 PM
Jay Z's proposal compared the average between *team* (or as i think should be *all*) players to determine how much PSR was gained/lost from match score, then added/subtracted a fixed win loss rate in addition to that. You have two independent values here that are added to make the total PSR gain/loss, which will zero sum out. The graph does not show this accurately as the PSR gains from a high match score game could infact make a player on the loosing team gain PSR.
E: Jay Z has modified his formula to account for All average Versus Team average, which makes the next segment unnecessary.
Next, this system (the system described by Jay Z) is Zero sum, as both halves of it are zero sum. The Simple Win/Loss is easy to see, if you win, +X PSR, if you Lose, -X PSR. The total PSR gain for the match would then be:
12*(+X)+12*(-X)= PSR change = 0
So this half is zero sum.
Now the next part is also zero sum, to explain we need to go over it all. we get the average match score of all players (A), then compare it to the Pilot score (P) to get the ratio between how well the pilot did versus the average person in the match. Then we Have to subtract one from this ratio to get the percent difference between the Pilot and the average, and if he is above or below that average player. This then is multiplied by a variable (Y) to give weight in accordance to how much PSR you want to potentially give based on match score. This will look like this:
((P/A)-1)*Y= PSR Change
Now, this looks like it could be non-zero sum, but that average pilot score is calculated from the game just played, so in the end, the calculation for all the 24 players would Sum all these changes up, one pilot at a time and all zero out. forums don't support symbolic summation so bear with me for just the first few examples and how we get there.
((P1/A)-1)*Y+((P2/A)-1)*Y+((P3/A)-1)*Y... = PSR change = 0(?)
Now it might not be obvious why this is at first, but what is the value A? It's the average of the Pilot scores. That means if you have *all* of the pilot scores being divided out the same way, we can substitute A for each of the Pilot score if we are calculating them all this way. this turns each of the factors into:
((A/A)-1)*Y+((A/A)-1)*Y...
So the summation becomes:
(((A/A)-1)*Y)*24 = PSR change = 0 (!).
(((1)-1)*Y)*24=0
((0)*Y)*24=0
So the Matchscore side of JayZ's suggestion is zero sum as well.
Some other things about this system, as you might note, is that it does not always give the same amount of PSR +/- each match. In very even matches, with no outliers of score, each player has a lower ratios as they are all closer to average, which means the total +/- of PSR given would be low (say only +15/-15), but a large stomp where players have very low or very high match score, their ratio is large as they are very far apart from average, and the +/- would be high (say +25/-25).
A quick way to show this is run this with two players who have 100 and 0 score in one game, and X = 20 you get +20 for the 100 score player, and -20 for the 0 score player a +20/-20. if they both score 50 however, we see they both "gain" +0 PSR, +0/-0 which is good as it indicates a even match.
Another part of this is that there is no parity between individual +/- of PSR, someone can get +20 PSR and nobody loses that much but there is a +0 and two -10 PSR players that balance him out, and this happens for all 24 players, so nobody could share a PSR +/- number and still zero sum.
So in the end, as Jay Z posted, the formula would be:
((P/A)-1)*Y + W*X = PSR change
And those letters mean:
P=Pilot Match Score.
A=Match Average score.
Y=Moddifier to how much Match score matters (e.g. 20). While somewhat arbirtrary, this is the amount of PSR gained by doubleing the average match score or lost by scoring 0 points
W=Win or lose modifier, +1 if win, -1 if lose.
X=Moddifier to how Much Winning or losing matter (e.g. 5). This is how much PSR you gain by winning or losing.
Edited by vipershark0, 17 June 2020 - 06:57 PM.
#14
Posted 16 June 2020 - 09:23 PM
MadcatX, on 16 June 2020 - 07:44 PM, said:
I hope other folks in this thread just simply vote. We get something in place and can adjust MS values later if an improvement in MM is not observed.
After reading through a lot of the other threads about this, I hope this can be a simple vote as well.
I like the core 1a best overall, but see 2a as a good alternate.
#15
Posted 16 June 2020 - 09:40 PM
Would like to see a modification of 2A to include that metric, if not, 1A is better.
I haven't done hours of research on this like many in the PSR discord have done. Please give Jay Z's proposal a thorough look.
#16
Posted 16 June 2020 - 09:44 PM
For matchmaking, it really does not matter if high performing losers do not gain PSR.
I'm not all that interested in specifics of match score. Kills and KMMD's could probably be worth more and Missiles Destroyed worth less, but that's about it.
Edited by Gagis, 16 June 2020 - 09:49 PM.
#17
Posted 16 June 2020 - 10:14 PM
I vote Core 2B with JayZ's modifications.
AFAIK this was refined in just the last few days with the help of Xiphias's analysis of PSR over time. (I havent kept up with the PSR Discord in detail as I have been busy in real life and realised that this had gone beyond the point that I could help much further)
Thankyou vipershark0 for your breakdown of Jay Z's proposal. It really explains it well.
(backup vote for Core 2A, which looks like my proposal from the other week)
Edit: update my Vote to 2C as described in detail in this post from Jay Z
https://mwomercs.com...ost__p__6338670
This version has a C variable that allows later tuning of how much win/loss or personal skill are weighted. And hence from 2C you can effectively generate 1B and 2B by adjusting the C variable.
Edited by Kamikaze Viking, 17 June 2020 - 07:17 AM.
#18
Posted 16 June 2020 - 10:20 PM
vipershark0, on 16 June 2020 - 08:51 PM, said:
I can't help but notice that if the primary factor is win vs lose (Core 2A and Core 2B) doing something like TKing 1 teammate and severely damaging another would SIGNIFICANTLY lower your teams chances of winning so doing it as a PSR-gain exploit would not work.
When the primary decider of whether you go up or down in PSR is whether you helped (or at least didn't prevent) your team from winning, the incentive to do "weird things" rather than win goes away.
I know everyone feels like "I got 1k damage, I solo'd 10 enemies, I should go up" is correct and not rewarding that is "unfair", but really, how much worse is staying exactly where you were for one game?
It's like the match just didn't count. Play another, do it again, win this time, and boom, you've got your massive PSR increase.
Nightbird did the math and other people simulated it. WLR is 33% accurate. MS/Jay Z's approach is only 5% (I think that was the number).
The 5% system "feels" better but that's only because we've lived with a system that demonstrably shoves everyone into Tier 1 over time and is also win/loss based. Core 2A/B are "bad" only by association.
I understand why everyone feels the way they do (I felt it too I'm in Tier 1 and shouldn't be), but try to see past the stigma of "that guy didn't help, why is he going up/not going down"?
He will go down. More often than he goes up.
I'm okay testing Core 1A, but please be open minded to Core 2A.
Edit:
Kamikaze Viking, on 16 June 2020 - 10:14 PM, said:
AFAIK this was refined in just the last few days with the help of Xiphias's analysis of PSR over time.
I'm not able to keep up on this as much as I'd like, I said MS/Jay Z's above, if the "Jay Z's" part of that is no longer accurate, please mentally replace "Jay Z's" with "Core 1's".
And if the math guys say 2B with Jay Z's modifications models better than 2A, I'm okay with that one instead of 2A.
Edited by Domenoth, 16 June 2020 - 10:27 PM.
#19
Posted 16 June 2020 - 10:37 PM
#20
Posted 16 June 2020 - 10:38 PM
Personally this is now a case of ........get on with it! We've essentially already discussed the pros and cons for the two core options and whilst both have their supporters and detractors, we are in danger of covering old ground.
Paul, you have identified your preferred system and been open to trying the other core options over time. This is great.
May I suggest you implement core1a, run it for a month, try out core2a for a month and so on? Hopefully we/you can then assess the results like for like. I would also support JayZ's zero sum modification to core2b option.
Whatever happens, may main hope is that it will revitalise both the game and my waning enthusiasm. Just, please, for the love of all that is stompy, get on with it!
Z.
8 user(s) are reading this topic
0 members, 8 guests, 0 anonymous users