283 replies to this topic

[Nightbird]

I am always ready to learn although I do not always like being taught. - Winston Churchill



As many of you know, there is a new PSR system to be implemented next week. PSR Community Version 1.0 based on Jay Z's suggestion. This prophecy is a look 200,000 matches into the future (around 2-3 months), on what can you expect from your MWO Quick Play experience post patch.

This prediction is based on a simulation. How the simulation works is a fairly involved process, you can read the details in the spoilers below.

Also, since I made my own suggestion (Nightbird's suggestion) for this update, I put the results for my suggestion next to Jay Z's for comparison. Both suggestions fall within the limitations imposed by PGI for this update. THERE IS SOME SLANDER OUT THERE THAT MY PROPOSAL IS OUT OF SCOPE, THIS IS FALSE, MY SOLUTION IS BOTH IN SCOPE AND IS EASIER TO IMPLEMENT THAN JAY Z'S SOLUTION.

Spoiler

The simulation is conducted as follows:

1. I pulled all the pilot data from Jarl's List, and Season 37 on the MWO Leaderboards. I used season 37 as a roster of likely pilots to play going forward, and increased the data for these pilots by including Jarl's data from past seasons. This final roster of 10k pilots is feed into the simulation.



2. To simulate the queue, I pull 24-96 pilots randomly from the roster, weighted by the games (frequency) they played (i.e. a pilot that played 10k matches has a 10x higher chance of being selected than a pilot that played 1k matches). This comes from Paul's statement there are up to 100 players in the queue. For now, I am using 25% for each of 24, 48, 72, 96 counts since I don't have further data.



3. The players in the queue are sorted by PSR, the highest 24 put into one match, the next 24 into the next match, etc etc.

4. To make each team, the 24 players in the match are sorted into two teams that minimize the PSR total difference between the two teams.



5. The match result and whether it is a stomp is determined randomly.



6. The MS earned by each player is randomized with the mean/center being their Jarl's average MS. From the match result and MS earned, I calculated Jay Z's PSR movement, and update each player's global PSR, along with win/loss count and other stats.



7. Repeat 200,000 times and output snapshots of data at every 25,000 interval.

The Prophecy

Post patch, what will Jay Z's PSR distribution look like? Here is a snapshot after 25k, 50k, 100k, and 200k matches.



As you can see, people with high and low average match score earned will not necessarily be sent to T1 and T5 the fastest. This is because PSR is not Skill, it is Skill*Games_Played. People who play more get sent to the extremes faster.

Why is this a problem? There is a wide skill band for any PSR level; as an example see the 200k graph with horizontal lines at the PSR=3000, 4000, 5000 levels showing a gap of 50-100avgMS for the same PSR value. The Match Maker (MM) needs accurate PSR values to make good matches, it doesn't have that here.


In one of PGI's threads, they talk about reducing stomps. Does that happen? In the graph below, I calculated the % stomps per 25k matches to see if Jay Z's proposal will improve this. For fun, I will throw in the simulated results from my suggestion as well going forward.



Surprisingly after 25k matches, Jay Z's system stops improving. I suspect that the benefit of the system ends after the most skilled and unskilled pilots who play the most often finish their movements. After that initial period, PSR confuses skill and games played leading to net 0 useful information provided by PSR. The result is no improvements to stomp reduction.


What about the quality of the teams made by the Match Maker, can we expect any improvements? I show here the win chance of teams made by the MM, with 50% being the ideal. The cumulative games # is the point in the overall simulation, but the bars displayed are only for the last 25k games to show if things are improving over time.



The bars show Jay Z's model improve a little over time, but overall because PSR is inaccurate, the MM has a hard time making matches. Overall there is no improvement over today.


Finally, what is the WLR distribution players can expect under Jay Z's system? The graphs below are cumulative after 25k, 50k, 100k, and 200k games. Please note that the bars at 0.3 and 2 includes the counts for players with WLR <0.3 and >2.



You can see over time that Jay Z's proposal will consolidate pilots towards 1 WLR, but the movement is minor.

Why is moving pilots towards 1 WLR important? Because if PSR is more accurately presenting people's skill over time, then the MM will make more games with 50/50 win chance, and pilots will move closer to 1 WLR as a result. This does not happen here.

Also, people enjoy playing near WLR=1, with players more like to quit playing the further they are from 1 for both the low and high directions. Overall there is no improvement over today.


The Conclusion
Is there a main reason that the community proposal doesn't work? The heart of the matter is that a "zero sum movement" of PSR is not the main goal of PSR. The main goal of PSR should only be to accurately present a pilot's skill to the Match Maker, and demonstrating that via mathematical evidence. Nothing else. The moment the discussion became separated from that, the result became inevitable.

Is the system I proposed the best approach? No. It is a stupid simple solution (from the KISS principle). There are far better solutions but they are comparably harder to present.



All men make mistakes, but only wise men learn from their mistakes. - Winston Churchill

Edited by Nightbird, 30 June 2020 - 09:40 AM.

[thievingmagpi]

a system that doesn't fit the scope of the proposed changes isn't a good system. Nor is a convoluted system which requires excess resource and specialized personnel to understand.

you failed to present a system that was appropriate with the tools, systems and personnel available. you don't have to keep acting butthurt. No one anywhere claimed that the other systems were flawless, but they fit within the scope of the proposal and can be easily and quickly adopted.

Edited by thievingmagpi, 28 June 2020 - 12:07 PM.

[Nightbird]

thievingmagpi, on 28 June 2020 - 12:06 PM, said:

a system that doesn't fit the scope of the proposed changes isn't a good system. Nor is a convoluted system which requires excess resource and specialized personnel to understand.

you failed to present a system that was appropriate with the tools, systems and personnel available. you don't have to keep acting butthurt. No one anywhere claimed that the other systems were flawless, but they fit within the scope of the proposal and can be easily and quickly adopted.

I presented how to calculate my system within scope. Also, a change to a system that doesn't accomplish any goals can never be good regardless of scope, right?

[Vindicated]

I still support you in that I believe W/L is the better system over time, but there probably would still be SOME improvement if poor (say 0.5) W/L T1 players get correctly reclassified as T5 (or at least not T1 if they aren't total cannon fodder). They won't get stomped by the players who should be T1. That said, as T1 (I shouldn't be but still) I am getting matches against cadets (I've even seen 3), so I hope the matchmaker is doing its best to balance the teams.

I went back to Jay Z's writeup google doc (relinked below for convenience from the proposal thread). He includes a divergence control portion (periodic partial reset). Unfortunately, that's more complex (but only relatively though) and I don't know if PGI will be willing to implement it. Maybe a seasonal or bi-seasonal full reset is the best they'd be willing to do, but not sure if that's fully better.

Jay Z, on 16 June 2020 - 06:54 PM, said:


CLICK LINKS FOR EXPLANATION IN DETAIL!!!!!!!!!!!!!!!!!!!
https://docs.google....3LgMBQJ4hE/edit

[Anomalocaris]

Appreciate your efforts Nightbird. I really do. Coming back after all this time away and putting in the time to craft this is admirable.

But in all reality, PGI is unwilling to take the steps to really improve. We're past the "fiddling while it burns" stage. Now we're just drawing in the ashes with pointy sticks. We'll sacrifice some lambs in the hopes that the gods will favor us, and nothing will really change in the end.

[MisterSomaru]

hm.... still doesn't fit with what parameters PGI gave. Also, I'm not sure why this is such an obsession to someone that never really plays solo queue, or has touched the game since soup queue was started.

[Nightbird]

I see that some who worked on the community proposal have posted, but none is stepped up to defend it. Why not put in a word like "your simulation is wrong, our proposal will make better matches and lower stomps" something like that maybe?

[Natural Predator]

thievingmagpi, on 28 June 2020 - 12:06 PM, said:

a system that doesn't fit the scope of the proposed changes isn't a good system. Nor is a convoluted system which requires excess resource and specialized personnel to understand.

you failed to present a system that was appropriate with the tools, systems and personnel available. you don't have to keep acting butthurt. No one anywhere claimed that the other systems were flawless, but they fit within the scope of the proposal and can be easily and quickly adopted.

A wild Ragnar Brown Sea post.


I know Birdy pretty well and I can tell you its not because he is "Butt hurt" that he's pointing it out. He genuinely loves Mech warrior and wanted to see it succeed. I discussed this with him yesterday and I found his conversation rational and without emotional or illogic. The mathematics is there, he not criticizing Jay Z for the system proposed; Nobody can criticize Jay Z commitment to Mech warrior games over time. He's simply taking a system and offering a better ending result based on mathematic result which after-all should that not be the goal?

[thievingmagpi]

Trashtier, on 28 June 2020 - 02:22 PM, said:

A wild Ragnar Brown Sea post.


I know Birdy pretty well and I can tell you its not because he is "Butt hurt" that he's pointing it out. He genuinely loves Mech warrior and wanted to see it succeed. I discussed this with him yesterday and I found his conversation rational and without emotional or illogic. The mathematics is there, he not criticizing Jay Z for the system proposed; Nobody can criticize Jay Z commitment to Mech warrior games over time. He's simply taking a system and offering a better ending result based on mathematic result which after-all should that not be the goal?

because it was pointed out many times why Nightbird's system was inappropriate and/or out-of-scope, and instead of adjusting he doubled down on it, then ran over to these forums with a highly inflammatory and out of context screenshot, which was very clearly attempting to defame the people working on the alternate solution.

"mathematical robustness" is irrelevant if the model fails to satisfy the terms laid out in the proposal. all the math and all the sound modeling in the world isn't going to make an improper system fit.

If PGI demonstrated actual, real and honest willingness to redesign their systems from the ground up, with the goal of achieving a Very Good result, rather than simply tweaking a couple of numbers, then sure, bring on what can do.

The passive aggression because no one was fawning over the models is just childish.

Edited by thievingmagpi, 28 June 2020 - 02:55 PM.

[Nightbird]

thievingmagpi, on 28 June 2020 - 02:53 PM, said:

because it was pointed out many times why Nightbird's system was inappropriate and/or out-of-scope, and instead of adjusting he doubled down on it, then ran over to these forums with a highly inflammatory and out of context screenshot, which was very clearly attempting to defame the people working on the alternate solution.

"mathematical robustness" is irrelevant if the model fails to satisfy the terms laid out in the proposal. all the math and all the sound modeling in the world isn't going to make an improper system fit.

If PGI demonstrated actual, real and honest willingness to redesign their systems from the ground up, with the goal of achieving a Very Good result, rather than simply tweaking a couple of numbers, then sure, bring on what can do.

The passive aggression because no one was fawning over the models is just childish.

Thank you for saying my system is mathematically robust and achieves a very good result. I will only disagree with the point my proposal requires PGI to redesign anything. As I stated here, you can do a rolling WLR calculation by setting PSR=WLR, after every game returning RESULT (1 or 0) from the game server for win or loss, and

new_PSR = {99 * [old_PSR/(old_PSR+ 1)] + RESULT} / {99* [1 - old_PSR/(old_PSR+ 1)] - RESULT + 1}

The number 99 creates a rolling average of 100 games.

There is no need to poll long term stats, everything is done with the systems in place today.

If a WLR around 1 with lots of decimals doesn't suit people (calculators don't really care), then re-scale it by multipling by 2500.

WLR=1.1 >> PSR=2750.

Tier 5 = [0-1500] Tier 4 = [1500-2000] Tier 3 = [2000-2500] Tier 2 = [2500-3250] Tier 1 = [>3250 no cap]

Edited by Nightbird, 30 June 2020 - 09:43 AM.

[KodiakGW]

Nightbird, on 28 June 2020 - 11:21 AM, said:

As many of you know, there is a new PSR system to be implemented next week.

Have they actually specified a date? June 30th, July 7th, or another date?

[Nightbird]

KodiakGW, on 28 June 2020 - 06:27 PM, said:

Have they actually specified a date? June 30th, July 7th, or another date?

yep

Paul Inouye, on 23 June 2020 - 05:12 PM, said:

[color=orange]UPDATE: ETA is Tuesday June 30th.[/color]

[GweNTLeR]

Could you please make some research on convergence speed per individual player? I like WLR approach, but It seems that it would take a lot of time to actually achieve some proper results.
Also, it would be great to account player rotation (new players income and retirement).

[General Solo]

Thank you for your efforts Nightbird

[Nightbird]

GweNTLeR, on 29 June 2020 - 12:21 AM, said:

Could you please make some research on convergence speed per individual player? I like WLR approach, but It seems that it would take a lot of time to actually achieve some proper results.
Also, it would be great to account player rotation (new players income and retirement).

I'm not sure what you mean? The graphs show that from the first 25k games, WLR outperforms, and keeps improving after that.

[GweNTLeR]

Nightbird, on 29 June 2020 - 06:20 AM, said:

I'm not sure what you mean? The graphs show that from the first 25k games, WLR outperforms, and keeps improving after that.

25k games in summary. I'm asking how long will it take for an individual to reach a somewhat stable PSR(WLR).

Edited by GweNTLeR, 29 June 2020 - 06:34 AM.

[Xiphias]

GweNTLeR, on 29 June 2020 - 06:33 AM, said:

25k games in summary. I'm asking how long will it take for an individual to reach a somewhat stable PSR(WLR).

If there are 25k game, 24 player per game, and a roster of 10k players that means on average, each player is playing 60 matches. Obviously, since it's weighted some players will play a lot more and some will play a lot less, but that's the community average match number.

[Nightbird]

The last graph I posted shows WLR move for the player base over time.

As Xiphas calculated, 25k games is 60 games per pilot average with variances on participation. 200K is already at 480 average, which is a lot of games.

The WLR system will use any additional information about the pilots to make better matches and push everyone closer to 1 WLR. It will never succeed, but as long as someone is not 1 WLR they'll move asymptotically towards 1. Player skill resists this movement, so the people less skilled will be <1, and people more skilled will be >1, and the separation between skilled and unskilled players will happen as long as the queue allows.

As for new account creation and retirement, they will throw ripples into the player base as they get moved the the correct WLR and slow down the convergence. That been said, with true WLR, it only takes 10 matches to throw an alt account into "T1".

Edited by Nightbird, 29 June 2020 - 07:04 AM.

[GweNTLeR]

Xiphias, on 29 June 2020 - 06:46 AM, said:

If there are 25k game, 24 player per game, and a roster of 10k players that means on average, each player is playing 60 matches. Obviously, since it's weighted some players will play a lot more and some will play a lot less, but that's the community average match number.

Well, first of all, noone says it finishes converging on 25k games. There are other reasons, but I'm not a native speaker so let's try to stay simple.

[Nightbird]

GweNTLeR, on 29 June 2020 - 07:00 AM, said:

Well, first of all, noone says it finishes converging on 25k games. There are other reasons, but I'm not a native speaker so let's try to stay simple.

The graphs show out to 200K games. May I ask what the comparison to is, when you say the convergence is slow?