713 replies to this topic

[Kodyn]

crazytimes, on 15 June 2020 - 01:32 AM, said:

We still arguing about how to sort the last few hundred people playing into different tiers so they can all play in the same match still?

By the looks of it, it doesn't matter how it's done, after the reset enough people will be butthurt and leave that it will have to be a full tier spread to get matches.

Just get the reset over with. The salt will flow whatever method is used thereafter, let's just get there.

Seriously. What this guy said.

I keep coming back to check on when the reset will occur, since it might prove interesting to play again for a bit. Played through the last match score event, but that was super easy and only took 2 days to finish, so I had hoped there'd be a more interesting event tied to the reset patch - or at the very least, get to enjoy a month or so of clawing my way back to T1, to see what the T1 pool actually feels like without valves opened for a short window - until they realize player count dictates opening them wide once more.

You know, those valves that PGI openly admitted to, that certain members of this forum swore never got opened despite others of us swearing for months and years that we'd been seeing potatoes in our matches...(Dead horse by now I'm sure, but I still find that last bit amusing. Especially considering the same community that didn't understand how the current PSR system worked is now designing the new one. The irony.)

[Mat Sorkas]

Kodyn, on 15 June 2020 - 02:31 AM, said:

Seriously. What this guy said.

I keep coming back to check on when the reset will occur, since it might prove interesting to play again for a bit. Played through the last match score event, but that was super easy and only took 2 days to finish, so I had hoped there'd be a more interesting event tied to the reset patch - or at the very least, get to enjoy a month or so of clawing my way back to T1, to see what the T1 pool actually feels like without valves opened for a short window - until they realize player count dictates opening them wide once more.

You know, those valves that PGI openly admitted to, that certain members of this forum swore never got opened despite others of us swearing for months and years that we'd been seeing potatoes in our matches...(Dead horse by now I'm sure, but I still find that last bit amusing. Especially considering the same community that didn't understand how the current PSR system worked is now designing the new one. The irony.)

You need to understand that with the current Tier system you have a lot of potatoes in T1. If someone played for a long time and never improved they still end up in T1 regardless of their actual skill level due to the upward bias of the PSR system. The matchmaker just does not have a tool to distinguish an ancient potato pilot from an actual battlefield god.
The whole conversation in this thread is about how to divide the playerbase into meaningful tiers. This will still not prevent "true T1s" from playing against weaker pilots. The number of concurrent players just is not there for it to be feasible. The valves would need to remain open to some degee.
However with a properly categorized players we could see teams that are more balanced so for every potato in your team there would be one in the opposing force.

[Eatit]

Mat Sorkas, on 15 June 2020 - 03:41 AM, said:

You need to understand that with the current Tier system you have a lot of potatoes in T1. If someone played for a long time and never improved they still end up in T1 regardless of their actual skill level due to the upward bias of the PSR system. The matchmaker just does not have a tool to distinguish an ancient potato pilot from an actual battlefield god.
The whole conversation in this thread is about how to divide the playerbase into meaningful tiers. This will still not prevent "true T1s" from playing against weaker pilots. The number of concurrent players just is not there for it to be feasible. The valves would need to remain open to some degee.
However with a properly categorized players we could see teams that are more balanced so for every potato in your team there would be one in the opposing force.

Changing the criterion the MM uses to sort teams is not on the table. The metric used to assign tier is what is being changed. The MM will continue to sort by size not tier.

The current MM combines Tier1, Tier 2, and Tier 3 for a normal match. It opens valves when that's not enough.

If you move people around in those tiers it will not make a difference. The same people will still be in each match and will still be sorted by size.

Balance will not change!!!

This is an attempt to placate the people who want to be seen as superior to others. They will rise to Tier 1 and be rewarded with that title. The title of Tier 1 will not be given to people who are less skilled but play a lot of matches.

This is an ego stroke for those that need to have their egos stroked. PERIOD

[OneTeamPlayer]

Zerex, on 14 June 2020 - 10:00 PM, said:

This is what happens in a team based PvP game when 100% f the rewards are based on the win.

How long will the player base hang round if this starts happening even in 10% of matches?

Also it doesn't matter if on paper Nightbirds system is perfect or not they haven't took the human factor into account. They will do anything to game the system.

This is entirely misleading as if this video was released in 2010 it either shows:

1. A battlefield populated with NPCs that took literally over a day in multiple instances to complete which was then tuned downward in difficulty due to player complaints, while simultaneously being overshadowed by advances in the ingame "tech" (higher levels, more legendary equipment, better consumables).

or

2. A private server, which is neither tweaked to the same standards as official servers nor the same item progression (characters would have gear that most players wouldn't attain by default or payment).

If you don't play wow in short this comment is like showing a 2013 release day unskilled Cataphract losing handily to a modern day fully skilled recently made clan mech (say a Corsair or whatever is most popular) and saying that represents a fair fight and/or the standard of the game.

Arguing in bad faith and terribly disappointing to see.

Ironically what the video does show is an example of how badly gameplay can suffer when a developer caves to player demands rather than using their own team to properly analyze and design with a side of the perils of not watching balance when new weapons are introduced thereby invalidating a large amount of previous content (sound familiar?).

Edited by OneTeamPlayer, 15 June 2020 - 04:41 AM.

[Xiphias]

Jochi Kondur, on 13 June 2020 - 06:25 PM, said:

Am I right in your assessment that in terms of the benefit to the player experience, Nightbird's solution >= Gagis' solution > Jay Z's solution. However, only Jay Z's solution fits within the scope of changes and therefore has the best chance of being implemented?

I'm not an expert in statistics, so I defer to those in the thread like Nightbird who are more knowledgeable, but generally yes, I think that's correct. Gagis' might be within the scope, haven't checked.

OZHomerOZ, on 13 June 2020 - 08:10 PM, said:

I read that other contenders who have remained within the scope/limits have the same problem of divergence which imo is easily fixed with regular reset's like some other games do, apparently, I read that somewhere on the forums.

Correct me if I am wrong.

Sure, divergence could be fixed with resets, however then you'd have bad matchmaking every time the reset happens. A better setup would be to have something like Solaris seasons (reset every three months) and just reseed (instead of reset) using the global stats for that period to place players in their appropriate tier. If that's on the table I would be all for using those numbers to get better placement.

Quote

What or who's goal is to win, match maker, solo players or groups?

A solo pilot does not have the same chance at winning as a group pilot.

If you make winning a large part of the formula it will favour grouped pilots and disadvantage solo pilots in terms of PSR movement. Is this really measuring a pilots skill is what I am saying.
Edit: Even Handsome Proton mentioned that maintaining a positive win loss as a solo was hard work or some thing close to it. Correct me if I'm wrong beautiful?

Groups don't need more advantages.

Match making is used to match pilots and groups with similarish skill to avoid hugh skillgaps between two teams to my knowledge.

Again correct me if I am wrong. amirite?

The goal is to balance matches, not to make players feel good about their ranking. If groups win more then the matchmaker would make it harder for them, if anything that disadvantages groups rather than advantaging them. I don't see why giving groups higher PSR is an advantage/bad. If they win more, they go up, it gets harder, so either:

1) They always play in a group and MM should consider them better (because groups advantageous to winning)
2) They inflate in a group and play solo. MM puts them higher than their skill and they lose, this drives down their record to where it should be.

Groups mixed with solos obviously messes with any match maker, but if getting close matches is the goal, groups that win more should be given a higher rank and be forced to play against either other groups or better players. People are getting two attached to the "player skill" term and putting player worth into the equation. Yes, a solo might be a better player than someone who always plays in a group, yet the solo might have a lower WLR, however when grouping for matches the group should absolutely be considered higher skill because they are more likely to win than the solo. What's better, 4 good solos or a group of 4 average pilots? If the solos are better they will win and more up, if the group is better then it will win and move up and be put against even better solos or other groups.

Moving up in tier isn't an advantage, it causes the MM to make things harder for you.

East Indy, on 14 June 2020 - 09:22 AM, said:

How many is "a bunch"? I'm curious how long it takes to get that divergence.

Also — while I acknowledge that a feature of Jay Z's proposal is a "fair" approach to contribution, what happens to long-term simulations when a coefficient (or whatever) is added to make significant PSR shifts for, say, the top 4 and bottom 4 only, and very tiny shifts to the other 16 players? In other words, can the system assume that two-thirds of the players it's being fed average 175-250 matchscore and work to keep them in the middle, sending only the really good and really subpar folks to the extremes?

Specifically, that image was with 100,000 total matches (matches per player 2240-2571) at smaller numbers the divergence is obviously less bad. Here are examples at
10,000 (193-293)

Spoiler

20,000 (415-554)

Spoiler

30,000 (633-807)

Spoiler

Basically, it's not too bad within a few hundred matches (several months for most players) and just progressively starts to get worse.

That said, the basic model that I put together isn't really something robust enough to be pulling hard statistical numbers about the game. It has a number of simplifying assumptions so it's more for illustrations purposes rather than anything else.

Edit: To your second question East Indy. I could play around with things and generate something like that, but as I said I don't consider the model robust enough for hard numbers. I'm making a variety of assumptions (such as how a player's actual match score varies around their average) that I don't really have the data for. To get good numbers you would need to pull in the real data and make fewer arbitrary assumptions (e.g. chances of winning being based on "secret skill" numbers).

I think yes though, if you move fewer people or reduce the total amount of PSR movement, the divergence will be slower. The downside is that slower divergence will also lead to accurate MM being slower if everyone starts in the same spot.

Edited by Xiphias, 15 June 2020 - 05:40 AM.

[Nightbird]

Xiphias, on 15 June 2020 - 05:34 AM, said:

Great sim, I have one suggestion though, take the number of matches each player plays based on the Jarl's data, and randomize the number of matches played in the simulation using that instead of saying everyone has an equal chance of playing matches.

Take one look on Jarl's on any page and you can see that the # games played can go from 100-500 to 10,000+. This has a major impact when PSR=skill*games played.

I would program this into the chance of being picked for a match, and make sure it gives a final matches played variance comparable to true data.

I didn't do in my sim because WLR isn't impacted by matches played, but MS random walks are severely impacted. I predict you'll see more of a of cloud between skill and PSR than the nice line you have right now.

This is intended to demonstrate that resets don't do anything.

Edited by Nightbird, 15 June 2020 - 07:52 AM.

[Surn]

While I like the evaluation of teams based on average skill, it does not address the definition of skill.

That is my focus, once a reasonable skill is agreed upon, the rest is easy.

[Zerex]

OneTeamPlayer, on 15 June 2020 - 04:40 AM, said:

This is entirely misleading as if this video was released in 2010

WoW Burning crusade was released in 2007, pretty sure by 2010 AV had been guttered of its NPC's and was just a base rush.

The point is if you make winning the only goal don't be surprised when players on play to win

Edited by Zerex, 15 June 2020 - 09:46 AM.

[Xiphias]

Nightbird, on 15 June 2020 - 07:33 AM, said:

Great sim, I have one suggestion though, take the number of matches each player plays based on the Jarl's data, and randomize the number of matches played in the simulation using that instead of saying everyone has an equal chance of playing matches.

Take one look on Jarl's on any page and you can see that the # games played can go from 100-500 to 10,000+. This has a major impact when PSR=skill*games played.

I would program this into the chance of being picked for a match, and make sure it gives a final matches played variance comparable to true data.

I didn't do in my sim because WLR isn't impacted by matches played, but MS random walks are severely impacted. I predict you'll see more of a of cloud between skill and PSR than the nice line you have right now.

First, thanks for the feedback.

There are a lot of things that could be done to improve the sim. It's a really basic model that I threw together to illustrate the point, rather than something that is going to be able to generate quantitative values. I'm using simulated players and then just picking from that list randomly to generate the overall matches, which I understand is a naive and not realistic assumption, but it made the code easy.

I suppose I could pull in real player data and the use that to weight the chances of playing a match or something similar to generate matches, but it would be more complicated than the simply model I'm using.

Another problem with my sim is that it's built on the assumption that matchscore accurately reflects skill. I based the skill values off of match scores from Jarls and I base the wins/scores in each match based this underlying skill value. Since they are directly linked, obviously matchscore will end up being a much better predictor than it actually is in game.

Another problem with that specific graph is that I used a generated random number that was evenly distributed rather than normally distributed to scale a player's "skill" into a match score for each match. You end up getting a much wider spread if you use the standard normal distribution.

Perhaps I'll go back in a few weeks when I have more free time and build a more robust simulator off the real game data. Could be a fun project and useful practice. If I did that would you be open to providing some quick feedback/input if I PM'd you on the forums?

[Nightbird]

Xiphias, on 15 June 2020 - 08:08 AM, said:

First, thanks for the feedback.

There are a lot of things that could be done to improve the sim. It's a really basic model that I threw together to illustrate the point, rather than something that is going to be able to generate quantitative values. I'm using simulated players and then just picking from that list randomly to generate the overall matches, which I understand is a naive and not realistic assumption, but it made the code easy.

I suppose I could pull in real player data and the use that to weight the chances of playing a match or something similar to generate matches, but it would be more complicated than the simply model I'm using.

Another problem with my sim is that it's built on the assumption that matchscore accurately reflects skill. I based the skill values off of match scores from Jarls and I base the wins/scores in each match based this underlying skill value. Since they are directly linked, obviously matchscore will end up being a much better predictor than it actually is in game.

Another problem with that specific graph is that I used a generated random number that was evenly distributed rather than normally distributed to scale a player's "skill" into a match score for each match. You end up getting a much wider spread if you use the standard normal distribution.

Perhaps I'll go back in a few weeks when I have more free time and build a more robust simulator off the real game data. Could be a fun project and useful practice. If I did that would you be open to providing some quick feedback/input if I PM'd you on the forums?

Add me on Discord and PMing me there will probably be better, after the MM is decided I'll likely hop off the forums.

I do have a suggestion on how to quickly program this without real data. For every player generate a number between 1 and 100 to represent their play frequency. When you are picking players randomly, do a double randomization. When one player is selected by the first randomizer, do a second randomization with the play frequency as a % to see if they truly join the team. Keep going until you have 24 players selected and this should give you the necessary impact.

As for whether avgMS = true skill, if you want to simulate that, you'd have to 1) use a bell curve to create a random population with true skill values, and 2) use a second randomization on each people to give their avgMS. (This avgMS will have 11% R-square predictive value compared to true skill). -- it's getting hard to describe so I'd skip this for now.

Edited by Nightbird, 15 June 2020 - 08:23 AM.

[Xiphias]

Surn, on 15 June 2020 - 07:38 AM, said:

While I like the evaluation of teams based on average skill, it does not address the definition of skill.

That is my focus, once a reasonable skill is agreed upon, the rest is easy.

For the purpose of matchmaking isn't the goal to balance wins/losses? If that's the case isn't skill the ability to win? Match maker doesn't care how you win as long as you do.

[Wraith of Shadow]

Xiphias, on 15 June 2020 - 08:18 AM, said:

For the purpose of matchmaking isn't the goal to balance wins/losses? If that's the case isn't skill the ability to win? Match maker doesn't care how you win as long as you do.

I thought the 'goal' was to populate both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.

[Capt Deadpool]

Agree W/L should be only effective measurement of skill. Everyone gets unlucky and has bad teams creating losses, but because we know this happens to everyone, any subjective feeling of, "I am so unlucky, don't punish me because of the potatoes on my team." is irrelevant.

Grouping reduces statistical likelihood of drawing a full team of taters, which is an option available to everyone, but there are also many people who enjoy playing solo. Hopefully phase two after, hopefully, most of the community and devs realize obviousness of basing PSR/MM around W/L will be balancing solos against groups via use of modifiers or multiple PSRs based on if someone is grouped or not.

[Capt Deadpool]

Wraith of Shadow, on 15 June 2020 - 11:02 AM, said:

I thought the 'goal' was to populate both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.

Well, the 'goal' is to find the most effective metric to base PSR/MM on, which is W/L (data/simulations should not even be necessary to grasp this), for the purpose of 'populating both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.'

[Nightbird]

Friendly question to people, what do you think a good MM is supposed to do?

Is it to bring unskilled players and skilled players as close to 1 WLR as possible? Or punish unskilled players with a very low WLR and reward skilled players with a high WLR?

Here's a simulation of the current MM graphing skill versus WLR





Here's a simulation of the WLR MM graphing skill versus WLR:




All suggestions that doesn't use WLR or avgMS will preserve the status quo. Vote what you want to see.

Edited by Nightbird, 15 June 2020 - 11:45 AM.

[Capt Deadpool]

^To rephrase, I'd say the purpose of an effective MM is to reward every player regardless of skill-level with competitive matches, with any sense of 'punishment' only being experienced by those who prefer stomps

Ideal world (which will never happen due to low population): top-tier comp world class Olympian 4-mans play against each other and have as close to 1:1 W/L as possible, and Day-1 Tier 5 headless chickens cadets who still have not discovered the mini-map should also be 1:1.

Maybe... current population of players being drawn from determines size of teams to better achieve competitive parity? 100 players online: 12v12, 75 players: 8v8, 50 players: 6v6...

[Laser Kiwi]

Capt Deadpool, on 15 June 2020 - 11:09 AM, said:

Grouping reduces statistical likelihood of drawing a full team of taters.

Oh there are plenty of teams out there that are a hindrance rather than an asset to the team, the fact potatos group together makes all this even harder. Sometimes dropping solo i'll see 4 guys with the same unit ON THE OTHER SIDE, take one look and be fairly sure i'm about to have a win.

[Xiphias]

Wraith of Shadow, on 15 June 2020 - 11:02 AM, said:

I thought the 'goal' was to populate both sides with players of approximately the same skill level, so that they can have a fairly even match up and not have one side dominate the other.

Yes, the goal is to get close matches. Obviously generating imbalanced teams to force everyone to a WLR of 1 isn't the goal. That said, how does the matchmaker get close matches? By building similarly skilled teams. How do you identify similarly skilled teams? By looking at the players and see how good they are at winning and then splitting them up as evenly as possible.

If all your matches are close then you will end up having a WLR close to 1. That's the goal, to balance wins/losses while matches teams as closely as possible. Apologies if that wasn't clear from my first post.

Another illustration using my simple simulation and Jay Z's model. 1 million matches (24k-24k per player) and the better team (highest combined skill) always wins (yes unrealistic assumptions, I know)

X (Win weight) Y (relative MS weight)
X = 5
Y = 0 (Blue, a W/L only system)
Y = 25 (Orange, a W/L and MS system)

You can see that the W/L system arrives at a good steady state with clearly defined tiers. Including the MS in the calculation causes the system to diverge and ends up putting things back to where they are right now. How long that takes will depend on the values used and the playerbase, but that's the eventual result.

[Nightbird]

Xiphias, on 15 June 2020 - 12:53 PM, said:

beautiful illustration, axis are avgMS horizontal and PSR vertical right?

Edited by Nightbird, 15 June 2020 - 02:04 PM.

[Kodyn]

Mat Sorkas, on 15 June 2020 - 03:41 AM, said:

You need to understand that with the current Tier system you have a lot of potatoes in T1. If someone played for a long time and never improved they still end up in T1 regardless of their actual skill level due to the upward bias of the PSR system. The matchmaker just does not have a tool to distinguish an ancient potato pilot from an actual battlefield god.
The whole conversation in this thread is about how to divide the playerbase into meaningful tiers. This will still not prevent "true T1s" from playing against weaker pilots. The number of concurrent players just is not there for it to be feasible. The valves would need to remain open to some degee.
However with a properly categorized players we could see teams that are more balanced so for every potato in your team there would be one in the opposing force.

I fully understand both how the current system works and the point of this thread...I do however think you missed my point. It's ok. I think anyone who thinks a PSR update can fix what's wrong with MWO is missing the point a bit as well. It's a start, but it may be too little too late, and without being coupled with a Faction Warfare revamp or something to draw people back in, I think it's a lot of wasted energy. By all means, numbers nerd it out to your heart's content, I just think there's a much bigger picture determining the effectiveness of any PSR system, and that it should be kept in mind. I've played other low population pvp games before, and after a certain point there's only so much you can do with matchmaking and PSR.