80 replies to this topic

[Kroete]

Simple model why tiers dont work with that low playercount:

We have around 10000 players,
lets use 3 tiers to make it easier,
so we have around 8000 average players and 1000 above and 1000 under.

Now lets asume every players plays 1 hour and they are divided evenly over the day,
means we have 333 average players and 41 each that are better or worse for every hour.
Now divide it by matchplayers (24) and you will see the problem with 3 tiers and 10000 players,
1 match each for the top and worse players and 13 matches for the average.
But we have not 3 tiers, we have 5, means no matches for t1 and t5 with out opening the valves.

A "flowing" matchmaker" is the only way for mwo with that low playercount.
Take all players aviable, put the best 24 together, then the last, then from the top again,
then the last again and so on.
Then its only balancing psr to match playerskill ...

Edited by Kroete, 20 November 2020 - 09:19 AM.

[ShiverMeRivets]

Nightbird, that is an interesting analysis, but I think your conclusion is wrong.
The top and bottom populations are fundamentally different from each other. None of them leave because their WLR is too high or too low, and thus, “fixing” their WLR to a value of 1 will only serve to hide the symptom.

So how are these populations different? Well this is just my speculation, but it is just as valid as your conjecture that “fixing” the MM to normalize their WLR will make them stay longer. My guess is that the low WLR sample are mostly very casual players that don’t take the game very seriously - they play for a while, then move to another game without being highly invested (or getting good). The high WLR sample are probably very competitive players. These kind tend to get highly invested in a game and then burn out. They then move to the next new thing to get invested in it. They don’t leave because they win too much. I suppose many (especially the younger ones) are in groups/guilds/clans that move together between games. It has nothing to do with their WLR.

Having said that, a MM that is able to achieve a tighter spread around WLR=1 is a good thing, and a good thing helps in retaining players.

[Nightbird]

ShiverMeRivets, on 20 November 2020 - 10:26 AM, said:

Nightbird, that is an interesting analysis, but I think your conclusion is wrong.
The top and bottom populations are fundamentally different from each other. None of them leave because their WLR is too high or too low, and thus, “fixing” their WLR to a value of 1 will only serve to hide the symptom.

So how are these populations different? Well this is just my speculation, but it is just as valid as your conjecture that “fixing” the MM to normalize their WLR will make them stay longer. My guess is that the low WLR sample are mostly very casual players that don’t take the game very seriously - they play for a while, then move to another game without being highly invested (or getting good). The high WLR sample are probably very competitive players. These kind tend to get highly invested in a game and then burn out. They then move to the next new thing to get invested in it. They don’t leave because they win too much. I suppose many (especially the younger ones) are in groups/guilds/clans that move together between games. It has nothing to do with their WLR.

Having said that, a MM that is able to achieve a tighter spread around WLR=1 is a good thing, and a good thing helps in retaining players.

Thanks for your feedback! There are many reasons for players to leave, be it not wanting to learn, getting stomped as a newbie, burning out, not having maps/features, not liking X. How do you determine the impact each of those items have? Well, you divide the population into groups of people that approve or disapprove of X, and compare their retention rates over time. The difference is the estimator of the impact of X.

As for the question of whether newbies are casuals that don't want to learn, or want to learn but get stomped, which is more correct?

And the question of whether more competitive players are bored because they win all the time, or are burned out from the effort, which better agrees with reality?

We don't know. It's not hard to ask questions no one can answer. But, in the end, do we base decisions on the data we have? Or go against data and hope the data is wrong? (I know you agreed the Matchmaker is important, I'm just using this as an opportunity to explain if someone else bring up X, Y, or Z that neither I nor anyone else can answer. It's a difficult topic.)

Edited by Nightbird, 20 November 2020 - 11:45 AM.

[Y E O N N E]

Nightbird, on 19 November 2020 - 08:45 PM, said:

The player proportion of each month that returns next month. So if of 100 players, 80 players return, the retention rate is 80%. Now, the next month may have 20 new players, meaning there are 100 players this month and next month, making delta in pop 0, but that doesn't mean retain rate is 100%.

Okay, so that's what I had in my head but didn't explain very well. Another conclusion I arrived at is that I was assuming the percentages were representative of the total playing population across your selected set of seasons, but that's not actually necessarily true. From the text of your test case, it sounds like what you are doing is comparing the list of players in Month n+1 against the list of players in Month n and then tallying them up in each WLR bracket, repeating that process for Month n+2 vs. Month n+1, etc. This would decouple your resulting chart from the actual size of the population playing MWO month to month, let alone over time.

I think if you really want to show the impact of WLR, and thus MM by association, on player retention to people who may not have the insight into the math, then a better way to do it might be to plot the median number of games played in each WLR bracket over time. If your theory is true, then the overall downward trend of the players at the ends of the WLR range should be more acute due to fewer players to play games at all and the few players remaining each playing fewer games. What issues do you see with this approach?

NOTE: I don't disagree with the premise that bad MM is driving players away. I actually agree with it; it's one of the primary drivers for players leaving, compounded by a steep learning curve, poor UI, and a myriad other issues that result from a lack of ongoing development. I just think that you can't quite yet draw that conclusion with what you have provided because it divorces the influencing trend from the impact on the big picture.

[Nightbird]

Miss Greene, on 20 November 2020 - 05:24 PM, said:

This would decouple your resulting chart from the actual size of the population playing MWO month to month, let alone over time.

You may be over thinking it. Each month only compares between that month and the next, using the full pop of that month.

Miss Greene, on 20 November 2020 - 05:24 PM, said:

I think if you really want to show the impact of WLR, and thus MM by association, on player retention to people who may not have the insight into the math, then a better way to do it might be to plot the median number of games played in each WLR bracket over time. If your theory is true, then the overall downward trend of the players at the ends of the WLR range should be more acute due to fewer players to play games at all and the few players remaining each playing fewer games. What issues do you see with this approach?

Retention rate by definition is by amount of people leaving or staying. You're welcome to graph your idea though.

Miss Greene, on 20 November 2020 - 05:24 PM, said:

I just think that you can't quite yet draw that conclusion with what you have provided because it divorces the influencing trend from the impact on the big picture.

What conclusion do you draw from the graph then?

[Y E O N N E]

Nightbird, on 20 November 2020 - 06:32 PM, said:

You may be over thinking it. Each month only compares between that month and the next, using the full pop of that month.

Maybe; that is what I said immediately after the quoted text. Yes.

Quote

Retention rate by definition is by amount of people leaving or staying. You're welcome to graph your idea though.

Leaving or staying...over what timeframe? What proportion of the timeframe does a player have to be absent before it counts as "lost"?

You've defined that time-frame as one month and the proportion as 100%, but you are sampling the game population over 36 months. To the health of the game, does it matter more if a player plays every other month religiously or if they play for three months and then never come back? The first case looks like it can totally bamboozle your counting methodology, but it will keep PGI's lights on. The second case will be easily counted, but will absolutely put PGI out of business.

The bigger issue with your counting methodology is that you can have a population that is steadily growing over time and still show a "retention rate" of less than 100% month to month. You can confidently claim that only some proportion of the player-base will play enough games to register on the board between any two months and that a particular subset of this portion is more likely to come back between any two months, but you cannot conclude that any of the players in your set have actually been lost to the game as a whole because your definition of retention is insufficiently robust. The slope of the total population trend will let you speculate that you have a lot of players coming in or going out, but you cannot make a correlation between change in population size and your "retention" because there are players month to month who don't get counted as "retained" because they aren't on the list of players from the previous month.

Ergo, you need to be showing the lifetime play activity of the players in each WLR across your 36-month range so you can say, yes, players with X WLR play less to the point of not coming back after they've played N number of games. Or, if you really want to drive home that the exodus is mostly because MM is making unsatisfying games, you show that gameplay has dropped off the most among the predicted groups over time.

I would love to plot it, but I am not going to puff up my chest and spin lame excuses to save face when I tell you that I won't: I don't know how to get the data and I don't honestly care enough to sink several hours into this project. Statistics was never my favorite subject, so I believe that you have the data to show the trend...I just don't think you are being effective at communicating it. You asked for the feedback, though, so there it is; dismiss it out of hand if you want, it won't bother me.

Quote

What conclusion do you draw from the graph then?

I can only draw the conclusion that players at the extreme ends of the WLR bracket are significantly less likely to come back between any two given months.

[Nightbird]

Miss Greene, on 20 November 2020 - 07:47 PM, said:

The bigger issue with your counting methodology is that you can have a population that is steadily growing over time and still show a "retention rate" of less than 100% month to month.

Just gonna point out that having a retention rate of over 100% is impossible by definition, so any argument that there is some logical flaw in not presenting retention rates over 100 is... flawed?

Edited by Nightbird, 20 November 2020 - 08:12 PM.

[Capt Deadpool]

Wow that is impressive work, and also surprising.

There are plenty of guys with >1.3 W/L who are definitely not seal clubbers, even beyond 2.0 W/L doesn't feel like seal clubbing... You are still losing 1 out of every 3 games at that ratio which is still quite a lot of losing. I would have thought both black and blue lines would experience/enjoy the satisfaction of winning (against similar tier opposition), which would lead to higher retention rates than evidenced by your graph.

I also think it depends on how 1:0 W/L s achieved, because I know I would be far more pissed off and play less if MM placed me in more games with more timid taters to reduce my W/L close to 1.0. But if I felt like both my team and opponent's were always evenly matched (very difficult to achieve with current population) so games were always close and even in losses the blue team behaved in a way that was conducive to winning reducing W/L to 1.0, I would play the same amount I currently am.

That the blue and yellow lines are nearly identical is the most surprising, as you have one group that is ostensibly prospering, and one that is not, yet they are 'enjoying' the game the same amount.

The graph proves your point so perfectly that I am almost skeptical, because I guess I subjectively value/enjoy winning more than the average black line/blue line player. If we can get every player to 1:1 without increasing stomps to achieve that (which I know is the goal), that sounds pretty perfect... imagine if every match there were only 1-4 players alive on the winning team.

Edited by Capt Deadpool, 20 November 2020 - 10:57 PM.

[VeeOt Dragon]

hmmm interesting (i also think MM is a big impact on player numbers) what i wonder is how much do the various events effect these numbers. i for one have all but given up on MWO. i still come back now and again if there is an event or something (mostly for Loot bag Events). i know of more than a few that are like me in that they only really return during these times.

2 things really need improving. first is the New Player experience (well that and some advertising so more people know the game still exists). and something to keep people playing after they get over that initial learning curve. the second could be mitigated but more maps and game modes so the same 2 maps aren't played over and over (hell i pickTerra Therma and Caustic just to get some variety. also it makes those who use over hot meta builds cry)

[RCore]

@nightbird thanks for the analysis. but one factor to consider - WLR is not an independent variable from a player's activeness (number of matches played).

WLR averages out the more matches you play, and will trend towards 1.0 for most people.

People that play very few matches tend to have a skewed high or low WLR. People that play very few matches are likely to be those that also won't play next season.

[Edeljoker]

Kroete, on 20 November 2020 - 08:03 AM, said:

As i said some month ago, during the matchmaker discussion,
doesnt matter, the valves will be open because too low players.



After some pause i tryied a few matches yesterday and dicrecly meet syncdropping groups.
As if 4 man are not enough to farm pugs?
Ill, maybe, try it in a few weeks again.
Maybe the syncers get bored until then?

Its just no fun anymore, game experience is at an all time low, IF you drop solo. And with group i had a ten games winning streak. And even that was boring.

Dropping solo just is a complete waste of time. Even if i make 4 kills and 1 k , i cant compensate for so many below 100 dmg players.

Its all just a dumb roulette now: first the roulette: " do i have the elite 4 man or not" and then the gameplay roulette called nascar.

If they dont get rid of the groups soon, this game may die BEFORE they bring the new changes.

Edited by Edeljoker, 21 November 2020 - 05:18 AM.

[Hunka Junk]

I am convinced that population is more of an issue than the matchmaker, population exacerbated by allowing group drops in QP.

So, now you are not only likely to encounter coordinated groups while playing solo, but they will be two tiers above you and throwing insults it must be your fault if their elite group fails.

This got dumber when the PSR AND the group drops were instituted at the same time. Competent game operators would implement them one at a time so that it would be clear how each affected play separately.

We don't have competent, so we don't know.

[The pessimistic optimist]

Hunka Junk, on 21 November 2020 - 06:01 AM, said:

I am convinced that population is more of an issue than the matchmaker, population exacerbated by allowing group drops in QP.

So, now you are not only likely to encounter coordinated groups while playing solo, but they will be two tiers above you and throwing insults it must be your fault if their elite group fails.

This got dumber when the PSR AND the group drops were instituted at the same time. Competent game operators would implement them one at a time so that it would be clear how each affected play separately.

We don't have competent, so we don't know.

^^ and some I mean some of those groups like quick play like this were they can sync and get easy stomps

Edited by SirSmokes, 21 November 2020 - 06:03 AM.

[martian]

Edeljoker, on 21 November 2020 - 05:14 AM, said:

Its just no fun anymore, game experience is at an all time low, IF you drop solo. And with group i had a ten games winning streak. And even that was boring.

Dropping solo just is a complete waste of time. Even if i make 4 kills and 1 k , i cant compensate for so many below 100 dmg players.

Its all just a dumb roulette now: first the roulette: " do i have the elite 4 man or not" and then the gameplay roulette called nascar.

If they dont get rid of the groups soon, this game may die BEFORE they bring the new changes.

Currently all I am seeing are premade groups farming PUGs. If you don't have such good premade on your side, you don't even have to play since the end result is obvious from the first minute.
My team starts every game with three players hiding in the corner of the map or behind buildings, or running around in singles or pairs. Every game ends up in defeat.

No point in playing this game.

Edited by martian, 21 November 2020 - 06:05 AM.

[Nightbird]

RCore, on 21 November 2020 - 04:58 AM, said:

@nightbird thanks for the analysis. but one factor to consider - WLR is not an independent variable from a player's activeness (number of matches played).

WLR averages out the more matches you play, and will trend towards 1.0 for most people.

People that play very few matches tend to have a skewed high or low WLR. People that play very few matches are likely to be those that also won't play next season.

First of all, great insightful comment!

On WLR averaging out towards 1 the more you play: If you go on the leaderboard for this season or any season, and sort by number of matches, you'll see that people have high or low WLR after playing hundreds of matches in a single season. Yes, it's true that if you toss a fair coin a few times, you see results further from 1 head/tails ratio due to having few attempts, and this will even out towards 1 as you add more attempts. If you toss a coin a few hundred times and get heads 30, 40% of the time though, you have a weighted (unfair) coin with 99.999% certainty. That is our Matchmaker today.

You're also right that people who play more games are more likely to stay than those that play few games, but isn't that putting the cart before the horse? Increasing retention rate is in essence the same goal as increasing the number of games people play, so comparing the two will show us that happy people are happy, unhappy people are unhappy. To improve the game, we have to graph levers we can pull against the happiness of the player base.

Edited by Nightbird, 21 November 2020 - 08:17 AM.

[Heffling]

How do you make a matchmaker that doesn't end up leaving players behind due to very high or very low w/l ratios (or match score or whatever)? Because you can definitely drive it to match top players only to top players, but if there isn't sufficient population, then the top players may end up with very extended wait times to get a match.

[Nightbird]

Heffling, on 21 November 2020 - 06:22 PM, said:

How do you make a matchmaker that doesn't end up leaving players behind due to very high or very low w/l ratios (or match score or whatever)? Because you can definitely drive it to match top players only to top players, but if there isn't sufficient population, then the top players may end up with very extended wait times to get a match.

I tried to explain this in another post: https://mwomercs.com...ost__p__6360543

To put it simply, a very skilled player on one team would warrant two skilled players on the other team for balance. There is no need to make anyone wait.

[Cluster Fox]

Sup Nightbird.

Nice data analysis.

BTW for those who doubt it: there's no assumptions from Jarl's numbers.
- A player is either in the jarl's list the next season or he isn't.

---

I'd like to add that the for anything to work, the MM needs to have some sort of secondary balance.

In all simulations, it's assumed the MM is trying to make both teams as equal as possible. My simulation used a simple yet effective flip-flop sort function. Nightbird's simulation used a more elaborate way to get a even better balance. What does the current MM use? My educated guess is... nothing.

Right now on most drops it seems like the MM just put a random arrangement of players that scored a match within it's MM criterion. The MM doesn't appear to make a single attempt at actually balancing both teams.

Now... there is no evidence it skips secondary balance. HOWEVER, from observations, if any kind of secondary balance takes place, it's so poorly done that it might as well not do it.

Maybe because mixed teams of groups and players was never planned, but that's a supposition.

There's no reason to have a 4 man group and 4 assaults on one side and pugs with 2 cadets and 1 assault on the other.

Edited by Cluster Fox, 22 November 2020 - 10:00 AM.

[Heffling]

Nightbird, on 21 November 2020 - 06:52 PM, said:

I tried to explain this in another post: https://mwomercs.com...ost__p__6360543

To put it simply, a very skilled player on one team would warrant two skilled players on the other team for balance. There is no need to make anyone wait.

Thanks, that makes sense.

[R5D4]

Nightbird, on 19 November 2020 - 05:58 PM, said:

Did I convince you? Leave your Yes/No and Why in a comment below!

No you did not convince me.

When I look at my own data for each time I've left MWO (for months or even years at a time) I can pretty much correlate each time with a specific change PGI made that I felt significantly impacted MWO gameplay in a negative way.

When I go back I see:
- UI 2.0/Skill Tree
- Poptarting Victors/Highlanders
- Small Pulse Laser Nerf
- Mech rescale (aka kill off all the lights!)
- LRM apocalypse
- Community Warfare let down
- Engine Desync
- Piranha Mech NERF BAT to the face!
- LRM apocalypse II: The dumbening
- Solaris - why are you making more bloody buckets when the population is already this small!?!
- etc...

When the change was reverted in a (PGI) short time I came back quickly and when it was never reverted (re-scale) I left for a LONG time.

Matchmaking is a problem yes but in large part it is a problem because the population has shrunk to the point that no matter how they "tune" it -- you just can't make lemonade from potatoes.