713 replies to this topic

[Jochi Kondur]

Gagis, on 13 June 2020 - 12:08 AM, said:

If their score is low enough, the Feelgood Factor will push their WinScore to the negatives. The Feelgood Factor was added so you would feel good about this.

What value should it be set to for that to happen? Alternating between 0 and 1 doesn't make a difference in the spreadsheet.

[Gagis]

Jochi Kondur, on 13 June 2020 - 12:25 AM, said:

What value should it be set to for that to happen? Alternating between 0 and 1 doesn't make a difference in the spreadsheet.

Ah, right. Should have written default values down so people can go back to them after someone has played with it. It gets compared to your Match Score so something between 50 and 150 might work. However, if the match is weird and the highest score anyone gets is 100 and FgF is 150, the PSR calculations go really wild. Thats why I'll probably recommend binding it to Average Score / 2.

[ghost1e]

Kurlon, on 08 June 2020 - 04:30 PM, said:

This system was developed by Jay Z with heavy input from the veteran community. A link to a full explanation is below;

https://docs.google....3LgMBQJ4hE/edit

In summary, this proposal uses simple arithmetic similar to existing calculations to assign fair and reasonable PSR shifts in accordance to pilot contribution to the match result. The aim of PSR is to segregate the playerbase on their respective match impact to give the Matchmaker the best possible information to create fair and interesting matches.

The version of this proposal you see presented is simplified from a more complicated model we have been testing which has more adjusting and weighting factors. Within the parameters set for this task, we believe this to be a straightforward implementation which will have community support. Finally, given there are only two values that control the maths, the long term management is simple and easy to hand over. If you have any questions, or concerns, please let us know as we have investigated this and numerous other solutions in great detail.

just one little thing I'd add, for a Tie just let W be 0 instead of 1 for a win and -1 for a loss

[Jochi Kondur]

Gagis, on 13 June 2020 - 12:30 AM, said:

Ah, right. Should have written default values down so people can go back to them after someone has played with it. It gets compared to your Match Score so something between 50 and 150 might work. However, if the match is weird and the highest score anyone gets is 100 and FgF is 150, the PSR calculations go really wild. Thats why I'll probably recommend binding it to Average Score / 2.

Ok, it works as advertised now. Cheers.

[Kamikaze Viking]

I vote For Jay Z's System
and Secondarily for my own (especially as tweaking is as simple as adjusting the values in the lookup table).

And for anyone asking. Jay Z actually built and tested mine in parallel with his and once we got it working, that was when this idea was put back on the table (im guessing it was just hidden along with others that were tested to remove clutter from his own presentation).

EDIT: Im now leaning towards voting for Gagis's sytem after nightbird and Xiphias have done their analysis.

Edited by Kamikaze Viking, 14 June 2020 - 01:51 AM.

[MechTech Dragoon]

Jay Z's system looks good. Go with it.

[Xiphias]

MischiefSC, on 12 June 2020 - 07:18 PM, said:

I don't play enough to be a real contributor to this discussion but I want to point this out -

Match Score doesn't equate to how effective someone is at winning matches. It's skewed by damage farming, kill stealing, etc. While top tier players often have a high average match score that's a product of a lot more than just damage numbers and component destruction.

The only useful metric for determining how effective someone is at winning matches is.... how often what they do, how they play, helps their team win matches.

So the question very honestly becomes:

Do you want a matchmaker that is as effective as possible at building teams who are evenly matched at winning

OR

Do you want a matchmaker that rewards the average players perception of 'I dun good' and punishes the average players perception of 'I dun bad'.

Maybe the game lacks the population to make the first super effective but the second is largely worthless as a matchmaker anyway. If the purpose of the new MM/PSR system is to just give player a sense of validation for 'I dun good' or 'I dun bad' that needs established. That's not necessarily a bad thing even; the community is a lot more 'stompy robbits EXPLOSIONS boom whee' than 'LEARN GIT GUD'.

I agree that W/L is the best predictor for determining W/L, however there is a significant contingent that disagrees with this (I think a lot of people are too focused on individual matches and not the bigger picture). That said, with the combining of queues, groups do have a potential to skew a W/L based metric, whether that really matters is a different question.

The way I see it, any good PSR system needs to includes some element of W/L (winning should increase your ability to climb), but a pure W/L metric could take a long time to converge to an appropriate level (and be skewed by teams). Looking at global stats, MS and WLR are correlated, players with better match score tend to win more. By including MS into the calculation you can increase the convergence rate (time to get to correct PSR) at the cost of introducing some steady state error (settling at a PSR that isn't exactly what WLR would eventually converge to). The top two players on the losing team are usually better than the bottom two players on the winning team, so the system moves players more quickly if they demonstrate consistent performance.

Given that T1-T3 players are already going to end up playing against each other when the valves are opened, I think you can make a decent argument that converging faster is worth sacrificing a bit of prediction accuracy. It also has the added benefit of appeasing players who dislike going down in PSR despite scoring well. No system is going to be perfect and I feel like a pure WLR system is unfeasible with the community opinion at the moment (old ELO system was based on WLR/ELO and people complained a lot about going down on a loss).

At the end of the day, a lot of players care more about the system rewarding them how they feel they should be rewarded, rather than how they mathematically should be shifted around to get the best matches. Lots of players care more about individual matches (e.g. going down on a loss with 8 kills 1200 damage) than overall trends.

[Jochi Kondur]

Kurlon, on 08 June 2020 - 04:30 PM, said:

- s n i p -

This system was developed by Jay Z with heavy input from the veteran community. A link to a full explanation is below;

https://docs.google....3LgMBQJ4hE/edit

And a spreadsheet simulating the system can be found here;

https://docs.google....dit?usp=sharing

- s n i p -

How-what-why is the difference between the values in columns R and S? Thanks.

Edited by Jochi Kondur, 13 June 2020 - 05:30 AM.

[Gagis]

Jochi Kondur, on 13 June 2020 - 05:28 AM, said:

How-what-why is the difference between the values in columns R and S? Thanks.

If I remember right he had one column where your score was compared to only your own team and another one where it was compared to both teams as two alternative ways to do it. Not 100% sure since it was an earlier version of his spreadsheet that I stole the layout from for my proposal.

[Jochi Kondur]

Gagis, on 13 June 2020 - 06:05 AM, said:

If I remember right he had one column where your score was compared to only your own team and another one where it was compared to both teams as two alternative ways to do it. Not 100% sure since it was an earlier version of his spreadsheet that I stole the layout from for my proposal.

Could you pass on the feedback to update the header descriptions to something more descriptive (the Little Details™...) Thanks!

[Nightbird]

Horseman, on 12 June 2020 - 10:54 PM, said:

You forget that some of us in this thread also have access to the discord. Your idea has been applauded by quite a few people. The problem they had with it was that it would be extremely difficult to convince most of the playerbase of its' value.

Normally the burden of proof is on the one proposing a solution to a complex mathematical problem. Going without it is the same as slapping down a 5 on a math test question without going through any steps to "show your work" and hope.

I already made an extra effort, not only to provide mathematical evidence that my solution is sound, but also that the proposed systems won't work. It's up to the rest of you that decide whether to leave the fate of the MM to sound evidence or to hope. If you all go with hope, then live with the results.

Edited by Nightbird, 13 June 2020 - 07:34 AM.

[Gagis]

Nightbird, on 13 June 2020 - 07:34 AM, said:

Normally the burden of proof is on the one proposing a solution to a complex mathematical problem. Going without it is the same as slapping down a 5 on a math test question without going through any steps to "show your work" and hope.

I already made an extra effort, not only to provide mathematical evidence that my solution is sound, but also that the proposed systems won't work. It's up to the rest of you that decide whether to leave the fate of the MM to sound evidence or to hope. If you all go with hope, then live with the results.

Dude, you have been awfully obnoxious about this. We all know, or at least I know, that your method would be good. But your method is also outside the scope of what was asked for. Unleashing a neverending barrage of messages on discord or a neverending barrage of posts on a forum won't make me believe in it any more than I already do. It will only make people annoyed and frustrated at being overwhelmed by sheer volume of noise.

And it certainly won't make PGI hire a developer to rebuild the system from the ground up either. You can't make that happen by repeating the same message over and over again.

My method might actually be outside the scope of what can practically be implemented too, but I went trough some extra effort to conform to the playerbase's wishes and the constraints of the existing framework, so I am somewhat optimistic it might work after all.

[Xiphias]

Nightbird, on 12 June 2020 - 07:30 PM, said:

The issue with the current PSR + MM is that you can calculate an average PSR movement per match for player, call it X, then the player's "skill" = # matches_played * X.

After 100 games, PSR=100X. After 2000 games, PSR=2000X. Whether the PSR really represents skill is convoluted with the number of matches.

It means you can have two players with PSR = 1000 and 3000, and the PSR = 1000 player reached it with 10 * 100matches, and the PSR = 3000 player reached it with 3 * 1000matches. It is clear the PSR = 3000 player is not as skilled. Such mix ups across the population means the Match Maker is blind when using this PSR.

With either PSR shift proposal above, we're changing X for Y. Is it obvious how much of an impact this will have on match quality?

Am I wrong in thinking that with a zero sum system the average PSR movement for a player will go down as they get matched against better players, resulting in the system leveling out? I.e. the best players will just continue to move up and max out, the worst will bottom out, but the rest will end up bouncing around in the middle? If you're performing relative to your team, only so many people can move up, so if you get paired with better players you will stop advancing?

The problem with the current system is that it's absolute, so that you can simply continue performing roughly the same and still move up. So in this case X is roughly constant, but Y is variable and decreases when competing against better players?

I haven't looking into the details of the math, but at a surface level it seems like that should hold. The main issue I can see with that is that if players drop out/new players join it could skew things over time.

[Tarl Cabot]

The one of the bigger issue I see between modifying the current PSR thresholds with a few levels vs Paul's proposal + others while working within the framework is that Paul's/others, within specific drop a player would be rewarded the same if Top Player in game XX with 220 MS would earn the same PSR points as Top Player in game ZZ w/413 MS.

Do we really want movement through the tiers to work that way, especially with the extremes of +24 vs -24? Or is it setup this way to make it more dynamic for said player while giving the MM something to work with?

Hmm, I think I may have answered my own question...

Edited by Tarl Cabot, 13 June 2020 - 09:26 AM.

[DevinMace]

Some people are too stuck on match score, wins do matter, sometimes the best way to get match score does not mean the best way to win. In the end it is about winning or losing, clearly we need more he that to make good match making though. However people suggesting winning shouldn't be weighed as much are well just wrong.

That being said it should all work it self out in the end, if you get a good match score regularly you will move up, people get too focused on small sample sizes.

There is no one with over 400 average match score with a losing record.

I think JayZ's method properly covers this, though I wish PGI would allow some variance in how much people gain or lose based on their personal PSR vs The other teams average psr. Much like how mot elo ladders work.

Sorry if some of this didn't make sense the keyboard I am using seems to not work every forth letter I type

[Nightbird]

Xiphias, on 13 June 2020 - 05:19 AM, said:

The way I see it, any good PSR system needs to includes some element of W/L (winning should increase your ability to climb), but a pure W/L metric could take a long time to converge to an appropriate level (and be skewed by teams). Looking at global stats, MS and WLR are correlated

You know the quote, "there are lies, damned lies, and statistics"? It is easy to produce stats that are wrong (and also maliciously to mislead people).

MS and WLR are correlated, but what you want to know is how well does past MS and past WLR predict future WLR? The answer is called goodness-of-fit in stats, or R-squared. MS has an R^2 = 12%, and WLR has R^2 = 37%.

To put it in laymans terms, you have two job offers, one is 50k/year, another is 150k/year. The community votes for the 50k job yes? (Sorry I lied, the current proposals are not as good as avgMS or Gagis's proposal, with the current community proposal being around 20k/year in comparative value with PGI's at 19k/year)

Gagis, on 12 June 2020 - 05:03 PM, said:

https://docs.google....t#gid=414734865

Details of my proposal are in the spreadsheet. Feel free to play with the values to see how it works.

I was wrong about your model not converging. I misread your adjustment on a loss. As for how well the model works, I would have to sim to find out but at this point I'm not willing to put in the work. I can sort of predict it though.

The question is, will this model work better than avgMS or WLR model?

Your model is an 'interaction' of MS and WLR effects. The problem with this is that MS and WLR are not just correlated, but they are collinear with each other. I'll leave this link if you don't know what collinearity is. https://www.britanni...rity-statistics

Because MS has higher variance than WLR and they are collinear with each other, the interaction will have the higher variance of MS mask the lower (beneficial) variance of WLR, therefore creating a model as good as avgMS, which will be 1/3 as good as the WLR model per the R^2 discussion above.

This will still be much better than Jay Z's proposal because it converges.

Xiphias, on 13 June 2020 - 08:21 AM, said:

Am I wrong in thinking that with a zero sum system the average PSR movement for a player will go down as they get matched against better players, resulting in the system leveling out?

Suppose Player A has true skill=1 and PSR=1000 and Player 2 as true skill=2 and PSR=2000, and they play together constantly, you would expect that Player 1's PSR can travel upwards but never approach 2000 where he would be bounced down. This is a misunderstanding because Player 2's PSR is not fixed at 2000, it is going up to 3000, 4000 etc. Therefore Player 1's PSR will not receive the downward influence you expect. There are also many other issues I listed in scattered posts.

The biggest problem with the current and community proposed systems is that despite PSR being supposed to a prediction of a player's skill, it will bounce up and down in a - random walk pattern - going off to the upper and lower caps, and fail to be anchored (converge) to distinct value presentation of skill between those caps.

This is why simulations are useful, it allows you to input the design of a system and see if anything was missed. I predict that 40% of the player base will (with enough games) reach the upper cap, and 50% will reach the lower cap. There will be 10% that forever bounce around in the middle.

The end result will be a system quantitatively equal to the current, and qualitatively worse. Quantitatively, the matches, the number of stomps, will not change after a retraining period with the new PSR. However, the people being sent to Tier 5 will be unhappy and more likely to quit the game.

Gagis, on 13 June 2020 - 07:59 AM, said:

Unleashing a neverending barrage of messages on discord or a neverending barrage of posts on a forum won't make me believe in it any more than I already do. It will only make people annoyed and frustrated at being overwhelmed by sheer volume of noise.

I'm passionate about stats, and I freely share what I know. That sort of results in what you see. That been said, I no longer play MWO and regardless of the results, I will not be harmed by the decision. Why don't I called it quits here, GGs.

Edited by Nightbird, 13 June 2020 - 11:01 AM.

[Kamikaze Viking]

Nightbird, on 13 June 2020 - 10:32 AM, said:

Because MS has higher variance than WLR and they are collinear with each other, the interaction will have the higher variance of MS mask the lower (beneficial) variance of WLR, therefore creating a model as good as avgMS, which will be 1/3 as good as the WLR model per the R^2 discussion above.

This will still be much better than Jay Z's proposal because it converges.

Thankyou. That's what we wanted all along. within the scope of the proposals that fit within PGI's scope which is the best.
If you are saying that your analysis says that Gagis's idea is better. then that's what we should get behind!

If you could elaborate for the masses WHY Gagis's is better that would be great

(feel free to comparitively shoot holes in mine too)

Edited by Kamikaze Viking, 13 June 2020 - 11:02 AM.

[ShogunKid]

Nightbird, on 13 June 2020 - 10:32 AM, said:

....
I'm passionate about stats, and I freely share what I know. That sort of results in what you see. That been said, I no longer play MWO and regardless of the results, I will not be harmed by the decision. Why don't I called it quits here, GGs.

I hope you don't quit in this discussion. Your reasoning and depth of understanding on this topic is a breath of fresh air around here. Your fair and objective comments are steering people to a correct way of thinking about AMS vs. W/L.

Best regards.

[Xiphias]

Nightbird, on 13 June 2020 - 10:32 AM, said:

You know the quote, "there are lies, damned lies, and statistics"? It is easy to produce stats that are wrong (and also maliciously to mislead people).

MS and WLR are correlated, but what you want to know is how well does past MS and past WLR predict future WLR? The answer is called goodness-of-fit in stats, or R-squared. MS has an R^2 = 12%, and WLR has R^2 = 37%.

I have some background in statistics, but I'm definitely not an expert. My personal view has consistently been that PSR should be based off WLR (I've read through your other posts on this years ago, good reads). I like WLR, if you play well, you win more, you move up. Easy enough. I don't have the time (or honestly the interest in the game) right now to pull all the statistics myself and run the numbers so I appreciate your much more knowledgeable insight

Having read the forums though I don't think a pure WLR system has the community capital to make it through the selection process. The old ELO system had this sort of problem where people complained a lot about getting stuck on bad teams and going down. People are far too focused on the individual matches and ignoring the larger trends (e.g. why should I move down when I scored 1200 damage?). Jay Z's simulator has this problem. It produces match results that look fair, but it doesn't project what scoring people that way will do to their PSRs over multiple matches.

Quote

Suppose Player A has true skill=1 and PSR=1000 and Player 2 as true skill=2 and PSR=2000, and they play together constantly, you would expect that Player 1's PSR can travel upwards but never approach 2000 where he would be bounced down. This is a misunderstanding because Player 2's PSR is not fixed at 2000, it is going up to 3000, 4000 etc. Therefore Player 1's PSR will not receive the downward influence you expect. There are also many other issues I listed in scattered posts.

Thinking a bit more about it I think I see the problem that you are illustrating. I was assuming that PGI would be hard capping PSR at some max for T1 (e.g. 5000), so those players would stop moving higher and then the players below them would run into the upper wall and stay on the edge of T1/T2, which would then bump the players below them to the edge of T2/T3, etc.

However, the problem then arises that the players at the bottom (0 PSR) can lose, but can't drop down any lower, so the players above them can effectively farm them by scoring higher and continue to get pushed up higher and higher inflating PSR while the worst players sit at the bottom without going down. Even if they got put on a team and won every match they would still go down if they had the lowest MS on their team.

Please correct me if I'm misunderstanding anything here.

As to your other posts, I read a few, but I'm not ambitious enough these days to read through nearly 30 pages of grey sea material so if I missed a few things, my apologies.

Quote

I'm passionate about stats, and I freely share what I know. That sort of results in what you see. That been said, I no longer play MWO and regardless of the results, I will not be harmed by the decision. Why don't I called it quits here, GGs.

Don't let a few hardheads on the forums get to you too much, though you're probably using your time better if you're not on here. Thanks for your insights and your discussion. I think you've convinced me of the flaws in a MS based PSR on top the drawbacks I already was aware of.

[Xiphias]

Nightbird, on 13 June 2020 - 10:32 AM, said:

The biggest problem with the current and community proposed systems is that despite PSR being supposed to a prediction of a player's skill, it will bounce up and down in a - random walk pattern - going off to the upper and lower caps, and fail to be anchored (converge) to distinct value presentation of skill between those caps.

This is why simulations are useful, it allows you to input the design of a system and see if anything was missed. I predict that 40% of the player base will (with enough games) reach the upper cap, and 50% will reach the lower cap. There will be 10% that forever bounce around in the middle.

The end result will be a system quantitatively equal to the current, and qualitatively worse. Quantitatively, the matches, the number of stomps, will not change after a retraining period with the new PSR. However, the people being sent to Tier 5 will be unhappy and more likely to quit the game.

So, against my better self interest, I went ahead a built a simple naive test simulator to illustrate the point for those here who don't understand the problem (like me until recently) with Jay Z's proposal (and ones like it).

Basic assumptions:

Spoiler

1) I used the global averages page of Jarl's list to get a rough mean/standard deviation to get a value for a player's "secret skill" and generated 1000 random players from this distribution.

2) I then assign each player a starting PSR of 1000 (cusp of T4), a tier, and a tracker for number of matches/wins
Tiers: T1 > 4k, T2 > 3k,T3 > 2k, T4 > 1k , T5 <= 1000

3) I select 24 unique random players from the list and balance the tiers as equally as possible

4) I use the normalized average of each team's "secret skill" to determine their chances of winning and then check this against a random number to determine which team wins.

5) I calculate a match performance for each player by multiplying their "secret skill" by a random value (I could also have potentially have used this in step 4 instead, but I wanted wins to be based on "secret skill" directly, shouldn't make a huge difference either way)

6) I used Jay Z's proposed system to generate PSR adjustments based on per team averages

7) I updated the PSR for each player and assigned a new tier as necessary

8) I ran a bunch of matches to get the final results of the system

Nightbird please correct me if I've made any egregious errors. T

This is by no means a robust statistical analysis, but just a quick model to show the problem with the proposed systems.

"Secret skill" vs PSR



Unsurprisingly, Nightbird was completely right in his assessment that Jay Z's system is going to end up giving us a similar problem to the existing system by effectively splitting the community into 2 main tiers, just like the current system (though with more players in T5 now).