Jump to content

Psr Update And Hold On Patch.


719 replies to this topic

#661 Xiphias

    Member

  • PipPipPipPipPipPipPip
  • Littlest Helper
  • Littlest Helper
  • 847 posts

Posted 15 June 2020 - 08:08 AM

View PostNightbird, on 15 June 2020 - 07:33 AM, said:

Great sim, I have one suggestion though, take the number of matches each player plays based on the Jarl's data, and randomize the number of matches played in the simulation using that instead of saying everyone has an equal chance of playing matches.

Take one look on Jarl's on any page and you can see that the # games played can go from 100-500 to 10,000+. This has a major impact when PSR=skill*games played.

I would program this into the chance of being picked for a match, and make sure it gives a final matches played variance comparable to true data.

I didn't do in my sim because WLR isn't impacted by matches played, but MS random walks are severely impacted. I predict you'll see more of a of cloud between skill and PSR than the nice line you have right now.

First, thanks for the feedback.

There are a lot of things that could be done to improve the sim. It's a really basic model that I threw together to illustrate the point, rather than something that is going to be able to generate quantitative values. I'm using simulated players and then just picking from that list randomly to generate the overall matches, which I understand is a naive and not realistic assumption, but it made the code easy.

I suppose I could pull in real player data and the use that to weight the chances of playing a match or something similar to generate matches, but it would be more complicated than the simply model I'm using.

Another problem with my sim is that it's built on the assumption that matchscore accurately reflects skill. I based the skill values off of match scores from Jarls and I base the wins/scores in each match based this underlying skill value. Since they are directly linked, obviously matchscore will end up being a much better predictor than it actually is in game.

Another problem with that specific graph is that I used a generated random number that was evenly distributed rather than normally distributed to scale a player's "skill" into a match score for each match. You end up getting a much wider spread if you use the standard normal distribution.

Perhaps I'll go back in a few weeks when I have more free time and build a more robust simulator off the real game data. Could be a fun project and useful practice. If I did that would you be open to providing some quick feedback/input if I PM'd you on the forums?

#662 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 5,218 posts

Posted 15 June 2020 - 08:16 AM

View PostXiphias, on 15 June 2020 - 08:08 AM, said:

First, thanks for the feedback.

There are a lot of things that could be done to improve the sim. It's a really basic model that I threw together to illustrate the point, rather than something that is going to be able to generate quantitative values. I'm using simulated players and then just picking from that list randomly to generate the overall matches, which I understand is a naive and not realistic assumption, but it made the code easy.

I suppose I could pull in real player data and the use that to weight the chances of playing a match or something similar to generate matches, but it would be more complicated than the simply model I'm using.

Another problem with my sim is that it's built on the assumption that matchscore accurately reflects skill. I based the skill values off of match scores from Jarls and I base the wins/scores in each match based this underlying skill value. Since they are directly linked, obviously matchscore will end up being a much better predictor than it actually is in game.

Another problem with that specific graph is that I used a generated random number that was evenly distributed rather than normally distributed to scale a player's "skill" into a match score for each match. You end up getting a much wider spread if you use the standard normal distribution.

Perhaps I'll go back in a few weeks when I have more free time and build a more robust simulator off the real game data. Could be a fun project and useful practice. If I did that would you be open to providing some quick feedback/input if I PM'd you on the forums?


Add me on Discord and PMing me there will probably be better, after the MM is decided I'll likely hop off the forums.

I do have a suggestion on how to quickly program this without real data. For every player generate a number between 1 and 100 to represent their play frequency. When you are picking players randomly, do a double randomization. When one player is selected by the first randomizer, do a second randomization with the play frequency as a % to see if they truly join the team. Keep going until you have 24 players selected and this should give you the necessary impact.

As for whether avgMS = true skill, if you want to simulate that, you'd have to 1) use a bell curve to create a random population with true skill values, and 2) use a second randomization on each people to give their avgMS. (This avgMS will have 11% R-square predictive value compared to true skill). -- it's getting hard to describe so I'd skip this for now.

Edited by Nightbird, 15 June 2020 - 08:23 AM.


#663 Xiphias

    Member

  • PipPipPipPipPipPipPip
  • Littlest Helper
  • Littlest Helper
  • 847 posts

Posted 15 June 2020 - 08:18 AM

View PostSurn, on 15 June 2020 - 07:38 AM, said:

While I like the evaluation of teams based on average skill, it does not address the definition of skill.

That is my focus, once a reasonable skill is agreed upon, the rest is easy.

For the purpose of matchmaking isn't the goal to balance wins/losses? If that's the case isn't skill the ability to win? Match maker doesn't care how you win as long as you do.

#664 Wraith of Shadow

    Member

  • Pip
  • Philanthropist
  • 12 posts

Posted 15 June 2020 - 11:02 AM

View PostXiphias, on 15 June 2020 - 08:18 AM, said:

For the purpose of matchmaking isn't the goal to balance wins/losses? If that's the case isn't skill the ability to win? Match maker doesn't care how you win as long as you do.

I thought the 'goal' was to populate both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.

#665 Capt Deadpool

    Member

  • PipPipPipPipPip
  • Ace Of Spades
  • 102 posts

Posted 15 June 2020 - 11:09 AM

Agree W/L should be only effective measurement of skill. Everyone gets unlucky and has bad teams creating losses, but because we know this happens to everyone, any subjective feeling of, "I am so unlucky, don't punish me because of the potatoes on my team." is irrelevant.

Grouping reduces statistical likelihood of drawing a full team of taters, which is an option available to everyone, but there are also many people who enjoy playing solo. Hopefully phase two after, hopefully, most of the community and devs realize obviousness of basing PSR/MM around W/L will be balancing solos against groups via use of modifiers or multiple PSRs based on if someone is grouped or not.

#666 Capt Deadpool

    Member

  • PipPipPipPipPip
  • Ace Of Spades
  • 102 posts

Posted 15 June 2020 - 11:13 AM

View PostWraith of Shadow, on 15 June 2020 - 11:02 AM, said:

I thought the 'goal' was to populate both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.


Well, the 'goal' is to find the most effective metric to base PSR/MM on, which is W/L (data/simulations should not even be necessary to grasp this), for the purpose of 'populating both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.'

#667 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 5,218 posts

Posted 15 June 2020 - 11:15 AM

Friendly question to people, what do you think a good MM is supposed to do?

Is it to bring unskilled players and skilled players as close to 1 WLR as possible? Or punish unskilled players with a very low WLR and reward skilled players with a high WLR?

Here's a simulation of the current MM graphing skill versus WLR

Posted Image



Here's a simulation of the WLR MM graphing skill versus WLR:

Posted Image


All suggestions that doesn't use WLR or avgMS will preserve the status quo. Vote what you want to see.

Edited by Nightbird, 15 June 2020 - 11:45 AM.


#668 Capt Deadpool

    Member

  • PipPipPipPipPip
  • Ace Of Spades
  • 102 posts

Posted 15 June 2020 - 11:31 AM

^To rephrase, I'd say the purpose of an effective MM is to reward every player regardless of skill-level with competitive matches, with any sense of 'punishment' only being experienced by those who prefer stomps Posted Image

Ideal world (which will never happen due to low population): top-tier comp world class Olympian 4-mans play against each other and have as close to 1:1 W/L as possible, and Day-1 Tier 5 headless chickens cadets who still have not discovered the mini-map should also be 1:1.

Maybe... current population of players being drawn from determines size of teams to better achieve competitive parity? 100 players online: 12v12, 75 players: 8v8, 50 players: 6v6...

#669 Laser Kiwi

    Member

  • PipPipPipPipPipPip
  • Shredder
  • Shredder
  • 242 posts

Posted 15 June 2020 - 11:49 AM

View PostCapt Deadpool, on 15 June 2020 - 11:09 AM, said:



Grouping reduces statistical likelihood of drawing a full team of taters.



Oh there are plenty of teams out there that are a hindrance rather than an asset to the team, the fact potatos group together makes all this even harder. Sometimes dropping solo i'll see 4 guys with the same unit ON THE OTHER SIDE, take one look and be fairly sure i'm about to have a win.

#670 Xiphias

    Member

  • PipPipPipPipPipPipPip
  • Littlest Helper
  • Littlest Helper
  • 847 posts

Posted 15 June 2020 - 12:53 PM

View PostWraith of Shadow, on 15 June 2020 - 11:02 AM, said:

I thought the 'goal' was to populate both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.

Yes, the goal is to get close matches. Obviously generating imbalanced teams to force everyone to a WLR of 1 isn't the goal. That said, how does the matchmaker get close matches? By building similarly skilled teams. How do you identify similarly skilled teams? By looking at the players and see how good they are at winning and then splitting them up as evenly as possible.

If all your matches are close then you will end up having a WLR close to 1. That's the goal, to balance wins/losses while matches teams as closely as possible. Apologies if that wasn't clear from my first post.

Another illustration using my simple simulation and Jay Z's model. 1 million matches (24k-24k per player) and the better team (highest combined skill) always wins (yes unrealistic assumptions, I know)

X (Win weight) Y (relative MS weight)
X = 5
Y = 0 (Blue, a W/L only system)
Y = 25 (Orange, a W/L and MS system)
Posted Image
You can see that the W/L system arrives at a good steady state with clearly defined tiers. Including the MS in the calculation causes the system to diverge and ends up putting things back to where they are right now. How long that takes will depend on the values used and the playerbase, but that's the eventual result.

#671 Nightbird

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The God of Death
  • The God of Death
  • 5,218 posts

Posted 15 June 2020 - 01:28 PM

View PostXiphias, on 15 June 2020 - 12:53 PM, said:

Posted Image



beautiful illustration, axis are avgMS horizontal and PSR vertical right?

Edited by Nightbird, 15 June 2020 - 02:04 PM.


#672 Kodyn

    Member

  • PipPipPipPipPipPipPipPip
  • Shredder
  • Shredder
  • 1,329 posts
  • LocationNY, USA

Posted 15 June 2020 - 02:00 PM

View PostMat Sorkas, on 15 June 2020 - 03:41 AM, said:

You need to understand that with the current Tier system you have a lot of potatoes in T1. If someone played for a long time and never improved they still end up in T1 regardless of their actual skill level due to the upward bias of the PSR system. The matchmaker just does not have a tool to distinguish an ancient potato pilot from an actual battlefield god.
The whole conversation in this thread is about how to divide the playerbase into meaningful tiers. This will still not prevent "true T1s" from playing against weaker pilots. The number of concurrent players just is not there for it to be feasible. The valves would need to remain open to some degee.
However with a properly categorized players we could see teams that are more balanced so for every potato in your team there would be one in the opposing force.



I fully understand both how the current system works and the point of this thread...I do however think you missed my point. It's ok. I think anyone who thinks a PSR update can fix what's wrong with MWO is missing the point a bit as well. It's a start, but it may be too little too late, and without being coupled with a Faction Warfare revamp or something to draw people back in, I think it's a lot of wasted energy. By all means, numbers nerd it out to your heart's content, I just think there's a much bigger picture determining the effectiveness of any PSR system, and that it should be kept in mind. I've played other low population pvp games before, and after a certain point there's only so much you can do with matchmaking and PSR.

#673 Xiphias

    Member

  • PipPipPipPipPipPipPip
  • Littlest Helper
  • Littlest Helper
  • 847 posts

Posted 15 June 2020 - 02:05 PM

View PostNightbird, on 15 June 2020 - 01:28 PM, said:

beautiful illustration, axis are avgMS horizontal and PSR vertical right?

Correct. Good practice would have been to label them, but I was lazy.

#674 50 50

    Member

  • PipPipPipPipPipPipPipPipPip
  • 2,088 posts
  • LocationTo Nova or not to Nova. That is the question.

Posted 15 June 2020 - 03:02 PM

View PostZerex, on 15 June 2020 - 07:39 AM, said:

The point is if you make winning the only goal don't be surprised when players on play to win


Not quite following your train of thought here.

#675 Vorpal Puppy

    Member

  • PipPipPipPipPip
  • Shredder
  • Shredder
  • 142 posts

Posted 15 June 2020 - 04:21 PM

It's too bad that PGI can't handle creating asymetrical teams (12 on red team, 9 on blue, for example). With our low population, that would help create more balanced matches.

#676 crazytimes

    Member

  • PipPipPipPipPipPip
  • Shredder
  • 396 posts

Posted 15 June 2020 - 05:00 PM

View PostMat Sorkas, on 15 June 2020 - 03:41 AM, said:

The whole conversation in this thread is about how to divide the playerbase into meaningful tiers.


... and then shove those exact same players into the same matches as before, just with different tier labels on them.

If everyone in this thread actually played the game instead of coming up with exotic maths to sort the same potatoes into the same match maker bucket, we wouldn't have as many issues with dying population.

#677 Kamikaze Viking

    Member

  • PipPipPipPipPipPip
  • Bad Company
  • Bad Company
  • 366 posts
  • LocationStay on Topic... STAY ON TOPIC!!!

Posted 15 June 2020 - 05:14 PM

@xiphias - if you have time, Ive sent you a PM on reddit with a PSR discord invite. We'd love to work with you on making this model more robust and integrating it into the proposal.
That visual is the best thing i've seen as a tiered distribution. Now all we need to do is work out how to implement it within the restrictions PGI have given. We have programmers and engineers but are lacking in a stats person, hopefully together we can crack this nut!

#678 Precise Nature of theCatastrophe

    Rookie

  • Bridesmaid
  • 1 posts

Posted 15 June 2020 - 06:03 PM

Will performance relative to players on your team count?

#679 Xiphias

    Member

  • PipPipPipPipPipPipPip
  • Littlest Helper
  • Littlest Helper
  • 847 posts

Posted 15 June 2020 - 06:24 PM

View PostKamikaze Viking, on 15 June 2020 - 05:14 PM, said:

@xiphias - if you have time, Ive sent you a PM on reddit with a PSR discord invite. We'd love to work with you on making this model more robust and integrating it into the proposal.
That visual is the best thing i've seen as a tiered distribution. Now all we need to do is work out how to implement it within the restrictions PGI have given. We have programmers and engineers but are lacking in a stats person, hopefully together we can crack this nut!

I went ahead and accepted the invite, but unfortunate I'm going to be extremely busy starting from tomorrow and running until the 26th so I won't have the spare time to talk about the details. If there's still a discussion after that, I'd be happy to jump on and try to talk through some of the details. If I end up having extra time before then I'll try and jump on and have a quick discussion.

Disclaimer though, I'm not a stats guy. I've had some background in stats, but it's definitely not my area of expertise. I was originally thinking some of these other models would work until Nightbird set me straight and I realized what the error was. I'm happy to do what I can, but don't take a quick simulator I threw together as a highly detailed knowledge of stats.

What I showed in the graph could be implemented in the current restrictions. It's just a simplified version of Jay Z's approach that only takes into account W/L and doesn't apply points for matchscore. I think the key here is that a player only moves up for winning and only moves down for losing. Otherwise the system is going to diverge over time. The challenge I see with the W/L is that it may take a long time to actually match people correctly. Based on my current understanding (subject to change), I think the best option right now would be as follows.

1) Make PSR change entirely dependent on W/L (Win you go up, lose you go down, players can also stay constant)
2) Reseed players based on historic data (PGI has this and did it with the introduction of the PSR system, could be based on AMS or WLR, but it should evenly distribute players into tiers of appropriate level)
3) If necessary increase PSR gain/loss within the team to more quickly move players (e.g. #1 MS on team gets +12, #12 MS on team gets +1, losing team is reversed #1 MS on team gets -1, #12 MS on team gets -12). This lets players feel like their contribution matters and moves better players more quickly, but (I think) shouldn't cause problems that break the PSR system.

It's a pretty straightforward system that shouldn't be hard to implement. Also, credit where credit is due, it's effectively identical to what Decency suggested recent and a long time ago.

I welcome corrections if I've made any errors in the above recommendations, but I think that might work to make a decent PSR system. The model I'm basing most of this off of is pretty simple and has a lot of assumptions (e.g. currently that the better team always wins), so it may not accurately reflect the game. I'd want to build a better model and do some more thorough work before committing to making any final suggestions.

#680 Zerex

    Member

  • PipPipPipPipPipPip
  • The Tip of the Spear
  • The Tip of the Spear
  • 298 posts
  • LocationUK

Posted 15 June 2020 - 11:07 PM

View Post50 50, on 15 June 2020 - 03:02 PM, said:


Not quite following your train of thought here.


On Incursion and assault base rushing to win the game to boost your PSR, in other words you might see a huge in games being won or lost with not a single mech dying, and in some cases, not even a point of damage being done.





2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users