26 replies to this topic

[Cluster Fox]

EDIT : I have reviewed all the information here and documented it through Google Docs.
Google Doc available here:
https://docs.google....dit?usp=sharing

This has now been simulated using season 48 data for validation. Post here:
https://mwomercs.com...psr-comparison/

Getting a stable and functional PSR is a two step process. However it is quite easy to achieve building on JayZ's system.

• Step 1. Modify the current X, Y, C.
  
• Step 2. Add a formula to stabilize PSR. Additional Input : PSR before the match - that's it.
  
• Step 3. New players start at 500 and there is no cap.

PRIORITY 1 : STABILIZE THE PSR

Uses a simple formula and PSR itself to make it stable. It doesn't really matter how the PSR shift is calculared at the end of a match for this to work.



PRIORITY 2 : X,Y,C TUNE UP
We need a PSR that fairly represents player skill across the player base so that the MM can rely on this information to balance teams. Otherwise, garbage in, garbage out.


Note : I discuss two other possible lines in the full document.
Answering the first question :

Why such a big X ?

Win and Loss is a hard +/-. You can’t 55% Win, you either 100% win or 100% loose. MS on the other hand changes smoothly and can take any value, mostly changing little by little. They behave very differently over multiple games.

Here's an example of how Win/Loss and MS act over many games. Note: there are many ways to get to the end values, but the end points do not change unless the resulting AvgMS and W-L changes.


Why are the X and Y important ?

Because the ratio between X and Y changes the steepness of the skill line. Too steep and you are MS heavy, like now, too shallow and it's all about WLR.

Next image shows how this fits together in my system. Movement along the PSR is represented by it's X and Y components in their own axis for clarity. Movement is always along the PSR line.


Can I test my end of match ?

Yes, but keep in mind that you can't look at a single match to see what PSR will do. To help with this, I added a portion where you can check the team composition if you were to play another match with the same players, based on the PSR shift alone. The tuned up X,Y,C provided will almost always split the winning and losing team.

Try it here with your own values on Jay Z’s spreadsheet (modified to show next match) :
https://docs.google.com/spreadsheets/d/1NewNKHTn4nPw9iMgoTIkIxoZYAxeM1YaI3m-hvX8LT8/edit?usp=sharing (Original X = 5, Y = 20, C=0.6)

https://docs.google.com/spreadsheets/d/15VQqOmtKDm9QinTQYZT_vBEu6upGV2qiWKCSVV99M1s/edit?usp=sharing (CFox Mod)

To know which PSR you would get, try the PSR calculator:
https://docs.google....dit?usp=sharing
Original post in the spoiler below - I got nothing to hide but the review made the analysis much better.
NOTE : Some of the findings are out of date.

Spoiler

Since finding posts in page X of an announcement is annoying, I'm moving the information here.

PGI has announced the Jay Z system will be implemented today, with the initially submitted parameters of X = 5, Y = 20 and C = 0.6

Jay Z's system is very versatile, but there are two challenges with the way PSR is updated.

• Choosing parameters that represent a player's skill accurately.
  
• Making sure PSR isn't a function of how many games a player plays (XP bar) & doing so within the "no player historical data" limitation set by PGI.

I'll copy my two other posts addressing both challenges here.

---

Challenge 1. Finding proper parameters.

==TL;DR;==

Using standard deviation. Best parameters for Jay Z's model are:

X = 0,7691 Y
Y = Any value. Depending on how aggressive PGI wants the PSR shift to happen.
C = 1

My suggestion, keeping the initial shift aggressiveness of Jay Z's model :

X = 11.53
Y = 15
C = 1

==Background==

I did some work on Jay Z's model. Since this is the only thing considered so far.

I still strongly support WLR based MM. However, I think we should try to make the best out of what is happening within the PSR limitations imposed.

The parameters X=5, Y=20 and C = 0.6 are arbitrary. I have not seen analysis behind those parameters to back them off. We need to find a way to rationalize the value of these parameters to make them represent true player skill.

==Baseline data analysis==

From season46&47 data:
Standard deviation for MatchScore is +/-57 MS
Standard deviation for WLR is +/-0.34 WLR (non linear)

Those two are linked as you may see on the data graph, so the linear skill progression line is one where the average WLR and the average Match score intercept.



The slope for this skill line is Y(avgMS) = 214.36 * X(WLR)

Knowing the distribution, Standard deviation PSR change for avgMS should be equal to standard deviation change for avgWLR.

This also means that a player on the top left corner of the greenish box (standard deviation box) is approximately the same Real Skill level than a player at the bottom right.
So, for example: Real Skill for 275avgMS & 0.68WLR ~= Real Skill for 161avgMS & 1.34WLR

Again, this is backed up by the standard deviation.

==Solving C, X and Y==
Variables:
W,X,Y,C,P,M,A as defined in the original post.

C = 1. The average split between losing and winning team avgMS is unknown. So we keep the MS shift independent from the overall avgMS and only relative to the team (W/L effect taken out of MS shift).

Jay Z's formula : PSR shift = W(X) - Y + Y( C(P/A) + (1-C)(P/M) )

C = 1 gives (1-C)(P/M) = 0

PSR = PSR(W/L) + PSR(MS)

To represent a movement along the axis, PSR(W/L),stdDev = PSR(MS),stdDev

W(X) = -Y + Y(P/A)

Plugging the standard deviation in P/A:

P/A = (218(+/-)57)/218 = 1.2615 or 0.7385

Since WIN is positive and loss is negative, the standard deviation is affecting the long term value of W, averaging to (WLR-avgWLR). Plugging the standard deviation in W:

W = (1.017(+/-0.34)-1.017) = +/-0.34

Putting all of this together we get:

0.34(X) = (1.2615 - 1)(Y)

Which yields:

X = 0,7691 Y

==Results==

The relation between Jay Z's model parameters to match historical data is:

X = 0,7691 Y

This ratio will result in player skill being represented along the line leading to 218 avgMS and 1.0165WLR. Other ratios between X and Y will either bias in favour of avgMS or WLR.

==Caveats==

WLR is non-linear. The linear approximation is accurate around avgWLR and avgMS which holds more than 70% of the player base.
Very small (<0.5) and very large WLR (>2.0) will be most affected by X. Underestimating and Overestimating Real Skill respectively. The bias is favourable since the PSR spread is increased and MM discrimination is also increased.

This conclusion applies for the current Jay Z's PSR formula only.

This will never result in a steady state value as the shift is only bound by artificial limits. This is a limitation of the PSR system as it has been discussed. I still strongly support WLR based MM.

Changing the amplitude of both X and Y will affect how fast the movement happens.

==Conclusion and suggestion==

My suggested values:

X = 11.53 (= 0,7691 Y )
Y = 15
C = 1

These values keep the initial aggressiveness, using the Standard deviation from season 46 and 47.

Now, if we set everything to 218(avgMS) in Jay Z's spreadsheet, and put one player at 275 and one at 161 for each team, we see an example of the shift for WLR and the relative shift for avgMS. (Col 2C)

---

Challenge 2. Stabilizing the PSR system
I used Jay Z's model for the base PSR shift, then tuned with my own values as seen in my last post.

==TL;DR;==

The PSR can be stabilized using a simple formula.

Only the player's PSR at the start of the game needs to be known, which the MM uses to balance a match.

==Known==

Jay Z's PSR shift system, using X = 11.53, Y = 15, C = 1

==Hypothesis==

Matchmaker knows a player's PSR when it's making a match, therefore it still has access to it at the end of the match.
- This cannot be verified unless access to the code is granted by PGI.
- This is the minimum data required to have a PSR system that stabilizes.

==Formula==
Once the match outcome is determined, we use Jay Z's model to calculate the PSRshift.

Then we transform it based on the PSR before the match.

PSRout = 2500 + (1 - P) * (PSRin - 2500) + (P * A * PSRshift)

or, in terms of a PSR change only. (I did not simplify, lazy)

NetPSRshift = 2500 + (1 - P) * (PSRin - 2500) + (P * A * PSRshift) - PSRin

Inputs

• PSRin is the PSR prior to the match
  
• PSRshift is the PSR shift calculated after the match (Jay Z)
  
• P is a response time, between 1 and 0. Small is smoother but takes more matches to represent skill.
  
• A is an amplifier. Equivalent to changing X and Y on Jay Z's system. This changes how spread out the distribution of players will be along the Y axis.

Note: A has the same effect as Jay Z's X and Y parameters, but allows to tune the PSR distribution as a whole rather than changing two parameters.






Outputs

• PSRout is the new PSR after the match
  
• NetPSRshift is the change after transforming. PSRout = PSRin + NetPSRshift

==Model==


5 archetypes - using a normal distribution for % because I'm no expert with the non-normal ones.

The potato : WLR and MS both 2 standard deviation below average. About 97.8% of players are stronger.
The weak : WLR and MS both 1 standard deviation below average. About 84.1% of players are stronger.
The average : WLR and MS both at the average. 50% of players are stronger.
The strong : WLR and MS both 1 standard deviation above average. About 15.9% of players are stronger.
The elite : WLR and MS both 2 standard deviation above average. About 2.2% of players are stronger.

Everyone started at 2500 PSR.

I did random end of games for each archetypes with the results varying around their average MS and WLR.
Jay Z's model was simply calculated against the averages.

The graphs have noise induced by W/L and MS differences and the trends are seen.

==Results==

Using Jay Z's system with the parameters mentioned, the amplifier A was set so that players over +/-2 standard deviations off the average hit the boundaries of 0 and 5000.

In both cases, A = 150



==Conclusion==

This formula can stabilize PSR to represent a skill level without having to fetch player information.

By taking known PSR before the match into account, we don't need artificial boundaries.

The formula can be adjusted for PSR to react quicker or slower by varying P.

The PSR distribution can be tightened around 2500 or spread out by varying A.

==Comparison - seeding A, not seeding and Jay Z only==

Edited by Cluster Fox, 14 October 2020 - 06:46 AM.

[Snowhawk]

That's the second time I read that they will make today the psr update. But I can't find any informations About this in the pgi announcements. Did you find it on Twitter?

[Cluster Fox]

Snowhawk, on 30 June 2020 - 06:57 AM, said:

That's the second time I read that they will make today the psr update. But I can't find any informations About this in the pgi announcements. Did you find it on Twitter?

Nope. Didn't see or read anything. I'm not too worried about it to be honest.

This is one of the most important changes to potentially reducing stomps and making the matches so much better. PGI taking their time to do it right is the best way.

[John Bronco]

It's literally the top post in announcements.

[Cluster Fox]

Very true, I read the question too fast. I haven't heard any updates about it today.

Edited by Cluster Fox, 30 June 2020 - 07:22 AM.

[Cluster Fox]

Answering three questions that were asked somewhere else:

---

Q1. What happens to the PSR if a player changes skill?

A. With the stabilizing PSR formula, Constant P determines how fast the PSR responds to change and how "noisy" it'll get. The onset stays the same regardless of when a shift is happening.

Here is an example of a fictional player with the "Weak" archetype for the first 200 games, then with the "Strong" archetype. Unlikely to happen but it illustrates the point well.
It took roughly 60 games to move down 1000PSRpoints from the initial 2500 PSR, and roughly 120 games to move up 2000PSR points.



For this example:
NetPSRshift = 2500 + (1 - 0.02) * (PSRin - 2500) + (0.02 * 150 * PSRjayz) - PSRin

I think P = 0.02 is a very reasonable response time/noise compromise.

---

Q2. So how what does the standard deviation mean in terms of MS and WLR, why is it equivalent?

A. To put it simply, standard deviation represents a proportion of players. So, for instance:
Player A: 40th percentile in MS and the 60th percentile in WLR
Player B: 60th percentile MS and the 40th percentile WLR

Both players are of equivalent skill, they simply progressed on different aspects of the game.

---

Q3. So where are the Tiers going to stabilize using your formula?

A. Since my formula uses the player distribution and standard deviation, we can actually plot an estimate on the graph as long as X=11.53, Y=15 and C=1 in Jay Z's system.

Changing parameter A will change how wide the bands are and tighten or spread everything around the average (mid Tier3). The lines are not perpendicular because the axes are not proportional.

Once in steady state, this is approximately how Tiers will be divided, assuming each Tier is worth 1000 PSR from 0-5000. The boundaries will always be blended due to noise.

For A = 150, boundaries will be more or less:



I personally should fall around low Tier2 and stay there unless I improve.

Edited by Cluster Fox, 01 July 2020 - 11:06 AM.

[DevilCrayon]

This looks promising! I don't understand all the statistics involved, but it sounds very plausible. I appreciate the time you put into it!

[Cluster Fox]

DevilCrayon, on 04 July 2020 - 07:47 AM, said:

This looks promising! I don't understand all the statistics involved, but it sounds very plausible. I appreciate the time you put into it!

Thanks, I know this is very dry. I'll try to explain it better in a Google Docs soon.

I'm in the process of changing WLR (W/L) "the X axis" to :
(Wins - Losses) / (matches played) -> This gives a scale between -1.0(all losses) and 1.0(all wins) and it's closer to how the PSR is added up.

It matches better with avgMS than WLR. Also it might change X,Y,C a bit.

I also extracted individual some season data in order to compare seasons. The original analysis was taking all the history of the s46 and s47 players.

[Sjorpha]

This model posits that doing things that does not contribute to the chances of winning should be regarded as some kind of "alternative" way of being good at the game. And even worse, it posits that this alternative way of being good should have equal weight in matchmaking.

How about if I said chess player A, that loses most of his games, is as good at chess as player B, who wins most of his games, because player A captures a lot more pieces in his games? Does that make any kind of sense? Nope obviously player B is better that player A, it's irrelevant how many pieces they capture.

You are good at a game if you're good at winning, you're bad at a game if you're bad at winning. MWO isn't some special case that is somehow exempt from game theory and competitive principles.

The purpose of the matchmaker is to predict your impact on the chances of winning a match, not to predict your matchscore or damage or anything else. Stop the nonsense please.

Edited by Sjorpha, 06 July 2020 - 05:05 AM.

[Nightbird]

I see that this is based on correlation between MS and WLR in one season. This is hindsight. You need correlation to future matches to have any predictive ability.

[Cluster Fox]

Sjorpha, I'm trying to improve on Jay Z's system since this is what got put in the game. I still support the WLR MM. I do see what you mean by considering the two as equal.

Nightbird, on 06 July 2020 - 06:38 AM, said:

I see that this is based on correlation between MS and WLR in one season. This is hindsight. You need correlation to future matches to have any predictive ability.

It's actually the full historical data of the players who participated in s46 and s47. I agree this is hindsight.

Edited by Cluster Fox, 06 July 2020 - 07:04 AM.

[Nightbird]

I mean try using season 46 data to predict season 47 results. Don't look within one season.

[Cluster Fox]

Nightbird, on 06 July 2020 - 07:10 AM, said:

I mean try using season 46 data to predict season 47 results. Don't look within one season.

I have only analyzed so far, I did not model to simulate. But thanks for clarifying.

I might try season 44 into 45 since 47 is weird because of mixed queue. 46 had about a month of mixed stuff.

[Cluster Fox]

So, let's forget how much WL and AvgMS is important for a second.

I documented my PSR stability solution. This one works no matter what PSR is. Check it out, it can probably be improved, maybe with a quadratic gain or something.

Google Docs:
https://docs.google....dit?usp=sharing

Edited by Cluster Fox, 09 July 2020 - 03:06 PM.

[Cluster Fox]

I reviewed my PSR skill line and documented it. W-L correlated much better than WLR.

Google Docs - Summary for both PSR mods with links to the -very- painful details:
https://docs.google....=h.rxz7ogwblorw

[Nightbird]

Take 1000 players from Jarl's with WLR=1. IMO they are as average skilled as one can be. They have avgMS from 175 - 300. Will you give them the same PSR value? Or will you give them PSR values from 1500 to 3500 messing up the Match Maker?

Stat has the saying "garbage in garbage out". If you take bad data such as match score, and do anything with it, the results will be bad. This is true of Jay Z's model, and since you're using it, your model as well.

Since we can't fix MS (not even if we change the kickers, it's beyond saving), then the next best thing we can do is ignore it.

Lastly, the PSR scale you're assigning to pilots are arbitrary. If someone has 3000 pts, are they 50% better than a pilot with 2000 pts? If the points don't mean anything, then it makes no difference when you sum two teams up to have the same value, since a 4000 pt and 2000 pt pilots added together will not be equal to 2 3000pt pilots.

Edited by Nightbird, 11 July 2020 - 10:24 AM.

[Cluster Fox]

Hey Nightbird, MS is not worth much right now, very true. But in my opinion, that doesn't mean it's completely worthless and to be ignored. Of course I might prove myself otherwise once I simulate this system.

To answer your question:
1.0 WLR and 100 AvgMS -> end value oscillates around 1800
1.0 WLR and 300 AvgMS -> end value oscillates around 2700
Not that bad of a spread on a range of 0-5000. There's always the option to make A smaller and tighten it.

The P-factor of 0.01 is high and induces noise right now. Seeing how a mess it is out there, PSR should get in the ballpark ASAP this season, noise is a compromise for now. A smaller P-factor doesn't change -where- values stabilize but it does on how fast they travel. So a P-factor of 0.003, once everything has kind of settled would make everything smoother and the PSR noise for a player would be reduced. Heck, maybe seeding P would be a good idea.

I disagree about the scale. The scale isn't arbitrary since it's based on standard deviation. So you know that a set percentage of player is above or below this PSR. A PSR of 3400 is 1 standard deviation both in W-L and AvgMS above the average. So about 16% of players are better. A PSR of 1600 is 1 standard deviation below average. 2500 is exactly average.
You can tell how much WLR and MS a player is likely to have if you know their PSR.

My proposition tries to make a distinction between those players who have the same WLR and are spread out in MS. I have not simulated it yet and maybe/probably the X,Y,C will change.

Oh and if I end up with Y=0 after simulating it, that basically made it a W-L system and you'll have full rights to "I told you so".

Cheers!

Edited by Cluster Fox, 11 July 2020 - 05:15 PM.

[Nightbird]

Cool, please sim Jay Z's system first and match the current results, then add your correction so we can see the difference in performance.

[Anomalocaris]

Hey look everyone, a rational discussion where the two proponents disagree but are looking for information and a solution rather than scoring points on each other.

Kill it fast before it spreads.....

[Nightbird]

Anomalocaris, on 12 July 2020 - 10:01 AM, said:

Hey look everyone, a rational discussion where the two proponents disagree but are looking for information and a solution rather than scoring points on each other.

Kill it fast before it spreads.....

The trolls are part of Cluster Fox's group, they're not gonna troll in their own member's thread lol