80 replies to this topic

[Nightbird]

As many of you know, PGI is taking suggestions on future updates for MWO and has compiled a large list of community ideas. However, PGI won't be able to implement every idea, the budget is limited after all. Therefore, how should it be decided which features will be implemented? The most popular ones? The most impactful? How do we decide? This thread shows a data-driven approach to identifying the most impactful feature.

To start, I downloaded seasons 17 to 51 leaderboard data from https://mwomercs.com...le/leaderboards . Players are categorized into 5 Win-Loss-Ratio (WLR) brackets: 0.9-1.1 is Green, 1.1-1.3 Blue, >1.3 Black, 0.7-0.9 Orange, and <0.7 red. Each bracket contains roughly 20% of the monthly population, the exact pop % in each bracket is labelled for each month. To calculate monthly player retention, we check to see if each player appears in the next month's leaderboard. If yes, then the player is retained, if no, then the player is lost. The graph shows the % player retention from each bracket for each month.



What this graph clearly shows is that players in the green bracket are the happiest of the five brackets, consistently, no matter what patch release or turmoil is happening that month. On the other hand, the Red and Black brackets, the worst and best performing players, are consistently the least happy of the five brackets. If we could put all players into the Green WLR bracket, we would get to use the highest retention rate for the entire playerbase.

There is only one feature responsible for which bracket a player falls in: the Matchmaker. Take my word as a pro that does this for a living, that it is easily possible for a Matchmaker to put ALL PLAYERS into the Green WLR bracket. The fact it doesn't, as shown by this graph, means that 50 out of every 100 players that left this game has done so because of the Matchmaker. If from the beginning we had a Matchmaker that put all players into the Green bracket, we would have a 50k playerbase today instead of 15k!

Since 50 out of 100 players already leave due to the Matchmaker, it is hard for another single feature to be more important wouldn't it? Therefore, the Matchmaker is king.

Did I convince you? Leave your Yes/No and Why in a comment below!

Some extra discussion:

Is using leaderboard data accurate?
There are issues such as alt accounts and player name changes. If PGI wanted to do this graph using hardware ID rather than pilot name, it would be more accurate. However I don't expect the overall numbers to change too much.

Can the Matchmaker really be improved?
This image from https://mwomercs.com...psr-comparison/ shows that using a bad WLR Matchmaker would put 60% of the pop into the green bracket, instead of 25% with our current Matchmaker. Explaining a good Matchmaker would put all of you into a coma so I'll pass on that.


Tracking next month for retention is too little, what if the player comes back afterwards?
Here is a graph showing retention as if a player returns in the next 3 months. The numbers are higher as expected but there is no difference in the effect of the Matchmaker.


What other features are impactful?
I compiled them here https://mwomercs.com...war-escalation/
They are marked with a (!), the rest marked with (o) are just my preferences.

Edited by Nightbird, 19 November 2020 - 06:11 PM.

[VonBruinwald]

It's almost as if clubbing seals drives players away...

[Absaint]

I didn't think that the matchmaker had a huge impact on player retention, was more inclined to other factors, such as new player experience.

But you convinced me of the utility of this approach.

This even impacts new player experience because they will fall into the green Win/Loss ratio even in their first days.

[The6thMessenger]

Lol. I'm at the green line. It's an interesting read, my take away is that that player-retention increases as WLR approaches 1.

So clubbing seals aren't helping with the player retention, who knew?

Sadly, I don't think the matchmaker would help at this point, mainly because of poor population. The new players will invariably have poor experiences either way due to being forced to match with vets anyways just to have a drop in a timely manner, and that will make us bleed players.

Edited by The6thMessenger, 19 November 2020 - 07:16 PM.

[Nightbird]

The6thMessenger, on 19 November 2020 - 07:14 PM, said:

So clubbing seals aren't helping with the player retention, who knew?

Both ends, hurts the seals more but the clubbers significantly as well.

[Y E O N N E]

What this shows is that, of all the players who return the next month, the majority of them have a WLR between 0.7 and 1.3. Which line do I follow to determine monthly percentage change in total population size? Because that's what the scale on the vertical axis indicates it's for, but I don't know what to look at to follow.

[Nightbird]

Miss Greene, on 19 November 2020 - 07:34 PM, said:

What this shows is that, of all the players who return the next month, the majority of them have a WLR between 0.7 and 1.3. Which line do I follow to determine monthly percentage change in total population size? Because that's what the scale on the vertical axis indicates it's for, but I don't know what to look at to follow.

This graph is just retention. The overall pop is the same as here: https://leaderboard.isengrim.org/stats

There are similarities in the shape, a big drop in pop will show a decrease in retention as well, but the important part is how the different WLR brackets perform within the same season.

[LordNothing]

VonBruinwald, on 19 November 2020 - 06:07 PM, said:

It's almost as if clubbing seals drives players away...

pgi never improved the solo experience beyond minimally viable product. a new match maker helps but qp is so dull mostly because it hasnt seen any real development beyond the addition of a handful of maps. attempts at new modes failed. attempt to draw solos into fp has failed, solaris failed. all those funds should have just been funneled into map development.

i also dont like the idea of the game deciding whether or not im going to win before the match starts.

Edited by LordNothing, 19 November 2020 - 07:51 PM.

[Anomalocaris]

Simple and well presented, and intuitive if you think about it. Few people, even dirty seal clubbers can be entertained for long without a challenge. Kind of makes the case that the matchmaker should be fixed before anything else is even tried. Even if it requires a substantial time or $$$ investment (although I'm inclined the believe you could halfway there with little investment).

Sure, go ahead and change back engine desync, get rid of ST heat spike, etc. But fix that matchmaker!!

[Y E O N N E]

Nightbird, on 19 November 2020 - 07:40 PM, said:

This graph is just retention. The overall pop is the same as here: https://leaderboard.isengrim.org/stats

There are similarities in the shape, a big drop in pop will show a decrease in retention as well, but the important part is how the different WLR brackets perform within the same season.

Okay, got it.

So what this is showing is:

• The proportion of each WLR group that came back each month as measured against the previous month
  
• The proportional composition of the total population across the set of WLR groups

Yes?

So my hang-up is that this doesn't seem to track with the total population size. The implication of your chart is that population count decreases every single month, which is not true when you look at the Jarl's chart. If my reading is correct (and it might not be), you would need to see spikes over 100% in your chart to account for those months where total population went up. The missing link could be the new players, which would slip past your test for the first month they played since you aren't looking for them in the previous month.

That, or the axis is just flatly mislabeled.

[Nightbird]

Miss Greene, on 19 November 2020 - 08:18 PM, said:

• The proportion of each WLR group that came back each month as measured against the previous month

The player proportion of each month that returns next month. So if of 100 players, 80 players return, the retention rate is 80%. Now, the next month may have 20 new players, meaning there are 100 players this month and next month, making delta in pop 0, but that doesn't mean retain rate is 100%.

[Thorqemada]

VonBruinwald, on 19 November 2020 - 06:07 PM, said:

It's almost as if clubbing seals drives players away...

It not only frustrates the clubbed Seals - it becomes boring also and frustrates the Clubbers too...

I said it once: Winning in itself is only a boring business!

Edited by Thorqemada, 19 November 2020 - 08:57 PM.

[The pessimistic optimist]

Nightbird, on 19 November 2020 - 05:58 PM, said:

As many of you know, PGI is taking suggestions on future updates for MWO and has compiled a large list of community ideas. However, PGI won't be able to implement every idea, the budget is limited after all. Therefore, how should it be decided which features will be implemented? The most popular ones? The most impactful? How do we decide? This thread shows a data-driven approach to identifying the most impactful feature.

To start, I downloaded seasons 17 to 51 leaderboard data from https://mwomercs.com...le/leaderboards . Players are categorized into 5 Win-Loss-Ratio (WLR) brackets: 0.9-1.1 is Green, 1.1-1.3 Blue, >1.3 Black, 0.7-0.9 Orange, and <0.7 red. Each bracket contains roughly 20% of the monthly population, the exact pop % in each bracket is labelled for each month. To calculate monthly player retention, we check to see if each player appears in the next month's leaderboard. If yes, then the player is retained, if no, then the player is lost. The graph shows the % player retention from each bracket for each month.



What this graph clearly shows is that players in the green bracket are the happiest of the five brackets, consistently, no matter what patch release or turmoil is happening that month. On the other hand, the Red and Black brackets, the worst and best performing players, are consistently the least happy of the five brackets. If we could put all players into the Green WLR bracket, we would get to use the highest retention rate for the entire playerbase.

There is only one feature responsible for which bracket a player falls in: the Matchmaker. Take my word as a pro that does this for a living, that it is easily possible for a Matchmaker to put ALL PLAYERS into the Green WLR bracket. The fact it doesn't, as shown by this graph, means that 50 out of every 100 players that left this game has done so because of the Matchmaker. If from the beginning we had a Matchmaker that put all players into the Green bracket, we would have a 50k playerbase today instead of 15k!

Since 50 out of 100 players already leave due to the Matchmaker, it is hard for another single feature to be more important wouldn't it? Therefore, the Matchmaker is king.

Did I convince you? Leave your Yes/No and Why in a comment below!

Some extra discussion:

Is using leaderboard data accurate?
There are issues such as alt accounts and player name changes. If PGI wanted to do this graph using hardware ID rather than pilot name, it would be more accurate. However I don't expect the overall numbers to change too much.

Can the Matchmaker really be improved?
This image from https://mwomercs.com...psr-comparison/ shows that using a bad WLR Matchmaker would put 60% of the pop into the green bracket, instead of 25% with our current Matchmaker. Explaining a good Matchmaker would put all of you into a coma so I'll pass on that.


Tracking next month for retention is too little, what if the player comes back afterwards?
Here is a graph showing retention as if a player returns in the next 3 months. The numbers are higher as expected but there is no difference in the effect of the Matchmaker.


What other features are impactful?
I compiled them here https://mwomercs.com...war-escalation/
They are marked with a (!), the rest marked with (o) are just my preferences.

Can't fix that if there is no one left to match you with

[Knight Captain Morgan]

Well done Nightbird. I'd be very interested to know if there is available data about average queue wait time. If so, I'd love to see a similar analysis of how average wait times affect retention. I'm guessing that when held up side by side, such analysis would reveal that the data shows matchmaking has the greater impact vs wait times.

Guess it's up to pgi to be transparent enough to tell. Maybe they don't want to admit the truth to themselves, much less to us. If they truly believe that letting groups club solos only caused "3%" more stomps then their grasp of reality may render such data moot anyways.

[Nightbird]

SirSmokes, on 19 November 2020 - 10:29 PM, said:

Can't fix that if there is no one left to match you with

Not an issue for 3 reasons.

1. We have 1/3rd the players from the start of the graph to today, yet the % of people in each WLR bracket hasn't changed. In other words, the size of the pop hasn't helped nor hurt the Matchmaker.




2. The Clusterfox graph you quoted shows that a simple WLR Matchmaker would put 60% of the pop into the Green WLR bracket.


3. If the previous two arguments don't convince you, then trust what a pro who does stats for a living says.

Knight Captain Morgan, on 20 November 2020 - 04:35 AM, said:

Well done Nightbird. I'd be very interested to know if there is available data about average queue wait time.

There isn't, we have access to the same data I'm afraid. I don't have any inside track to stuff.

Edited by Nightbird, 20 November 2020 - 07:48 AM.

[The pessimistic optimist]

Nightbird, on 20 November 2020 - 07:26 AM, said:

Not an issue for 3 reasons.

1. We have 1/3rd the players from the start of the graph to today, yet the % of people in each WLR bracket hasn't changed. In other words, the size of the pop hasn't helped not hurt the Matchmaker.




2. The Clusterfox graph you quoted shows that a simple WLR Matchmaker would put 60% of the pop into the Green WLR bracket.


3. If the previous two arguments don't convince you, then trust what a pro who does stats for a living says.



There isn't, we have access to the same data I'm afraid. I don't have any inside track to stuff.

*sigh* still not getting it you are making way to many assumptions

[Kroete]

The6thMessenger, on 19 November 2020 - 07:14 PM, said:

Lol. I'm at the green line. It's an interesting read, my take away is that that player-retention increases as WLR approaches 1.

So clubbing seals aren't helping with the player retention, who knew?

Sadly, I don't think the matchmaker would help at this point, mainly because of poor population. The new players will invariably have poor experiences either way due to being forced to match with vets anyways just to have a drop in a timely manner, and that will make us bleed players.

As i said some month ago, during the matchmaker discussion,
doesnt matter, the valves will be open because too low players.

Thorqemada, on 19 November 2020 - 08:56 PM, said:

It not only frustrates the clubbed Seals - it becomes boring also and frustrates the Clubbers too...

I said it once: Winning in itself is only a boring business!

After some pause i tryied a few matches yesterday and dicrecly meet syncdropping groups.
As if 4 man are not enough to farm pugs?
Ill, maybe, try it in a few weeks again.
Maybe the syncers get bored until then?

Edited by Kroete, 20 November 2020 - 08:09 AM.

[bilagaana]

The question remains: What would be the minimum active player population required to support an ideal matchmaker--one that would (unlike the current algorithm) consistently and accurately form balanced teams exclusively within set skill levels?

My guess is: A whole lot larger than the population we have.

The above is a serious question, by the way. I'm wondering what the math would indicate.

[Nightbird]

bilagaana, on 20 November 2020 - 08:35 AM, said:

The question remains: What would be the minimum active player population required to support an ideal matchmaker--one that would (unlike the current algorithm) consistently and accurately form balanced teams exclusively within set skill levels?

My guess is: A whole lot larger than the population we have.

The above is a serious question, by the way. I'm wondering what the math would indicate.

Making a Matchmaker better than the one we have today is possible with a population of 24 online players in queue. Ideally, 48 or 72 would work better, and we clearly have more than that today outside of a few hours during the lowest pop periods. Obviously more players than that helps even more.

Without going into math, first the Matchmaker should identify the skill of each player and shuffle players between teams until the teams are balanced, which is a 50/50 win chance for each team. That 40% of players have a WLR <.7 and >1.3 shows we're not doing that today. Second, the Matchmaker should make the skill gap - the distance between the highest and lowest skilled played on each team - as narrow as possible. This means when there are 48 players, the top and bottom 24 players should be put into their own matches. When there are 72 players, you can divide into 3 games. The more players there are, the smaller the skill gap in each team. We're not doing that either.

Edited by Nightbird, 20 November 2020 - 08:56 AM.

[Knight Captain Morgan]

bilagaana, on 20 November 2020 - 08:35 AM, said:

The question remains: What would be the minimum active player population required to support an ideal matchmaker--one that would (unlike the current algorithm) consistently and accurately form balanced teams exclusively within set skill levels?

My guess is: A whole lot larger than the population we have.

The above is a serious question, by the way. I'm wondering what the math would indicate.

It is quite mathematically possible. First come up with a formula to assign a score to each players. Average match score, win loss ratio, combination of factors, but something far more accurate than a 1-5 tier system. Then take ANY 24 players and sort them between red and blue in a manner that gives the smallest possible numeric difference when the 2 teams’ cumulative player scores are subtracted from one another.