80 replies to this topic

[Nightbird]

R5D4, on 23 November 2020 - 07:54 AM, said:

When I look at my own data for each time I've left MWO (for months or even years at a time) I can pretty much correlate each time with a specific change PGI made that I felt significantly impacted MWO gameplay in a negative way.

PGI updates can make all the retention lines go up or down, but it doesn't change the fact the WLR bracket plays a consistent role in predicting retention over the past 3 years right? Checking your stats on Jarl's, you're in the black bracket most of the time. This model shows that you're in the second worst bracket for retention, with the rate plummeting below 50% after certain updates.

While looking at the pop chart, you can argue that certain items, like Solaris, had a large impact for a month or two. However the impact of the Matchmaker has added up month by month, year by year, that it overwhelms the impact of any other single feature request. As far as cost is concerned, a good Matchmaker costs less than a single map, so it's hard to understand why this issue hasn't been addressed to date.

R5D4, on 23 November 2020 - 07:54 AM, said:

Matchmaking is a problem yes but in large part it is a problem because the population has shrunk to the point that no matter how they "tune" it -- you just can't make lemonade from potatoes.

I've made several posts on this already, so feel free to write a specific objection. The MM we have today can be described as 1 step up from having no Matchmaker, or 9 steps down from the best Matchmaker. Our pop is more than enough to permanently fix it. (Check out the last section of the first post, or the last post on page 1)

Edited by Nightbird, 23 November 2020 - 12:06 PM.

[R5D4]

Nightbird, on 23 November 2020 - 08:33 AM, said:

Checking your stats on Jarl's, you're in the black bracket most of the time. This model shows that you're in the second worst bracket for retention, with the rate plummeting below 50% after certain updates.

I’m probably in the black bracket all the times except for when PGI applies gameplay/balance changes that negatively impact the game lol.

I also play the light weight class and solo quickplay pretty much exclusively these days. So I would say I’m highly sensitive to gameplay/balances changes.

Nightbird, on 23 November 2020 - 08:33 AM, said:

While looking at the pop chart, you can argue that certain items, like Solaris, had a large impact for a month or two.

I would argue it had an immediate dramatic impact followed on by a long term impact (both Solaris and Faction Play) because these game modes further leached away the population from a centralized source (quick play) and split them up into multiple buckets none of which interact together. This is largely why I don’t give the idea of fixing the matchmaker much credence given that PGI has made it SO much more difficult to pull together a proper team from the outset.

You are probably better equipped to do the math than I am as this is not my forte but when you have a population of (lets generously say) 900 people at any one time playing but 250 of those are in faction play, 450 are in quickplay (split between europe, north america, and australia) , and 148 are in Solaris and 52 are in private matches with an uneven distribution spread between the different tiers (say more low level players are in quickplay than top tier players who are in faction play and private) how is the Matchmaker going to make a “good match” from that?

What even constitutes a “good match” at that point? I guess you could say a “good match” is one where most of the players end up coming out with a near 1.0 win loss ratio but that’s not how I personally measure it.

I play to do my best performance, some days that’s me trying to increase my k/d ratio, some days that’s me trying to do the highest match score in the smallest mech with the least armor. Almost never is it me trying to increase my win/loss ratio because in quickplay solo in a flea or a locust that just isn’t going to happen.

Nightbird, on 23 November 2020 - 08:33 AM, said:

However the impact of the Matchmaker has added up month by month, year by year, that it overwhelms the impact of any other single feature request. As far as cost is concerned, a good Matchmaker costs less than a single map, so it's hard to understand why this issue hasn't been addressed to date.

I've made several posts on this already, so feel free to write a specific objection. The MM we have today can be described as 1 step up from having no Matchmaker, or 9 steps down from the best Matchmaker. Our pop is more than enough to permanently fix it. (Check out the last section of the first post, or the last post on page 1)

Matchmaker is certainly the most consistent presence but is it really the determining factor? I had a group of 300+ people I would play with back in the hay day of MWO and yes we all (every one of us) complained about the matchmaker but the things that drove them away (so far as I heard) were things like the rescale, engine desync, stupid balance changes that made zero sense, etc…

I agree that improving the matchmaker is worthwhile endeavor as at least it might help stop/slow down the bleed of new players. I just don’t think improving it is going to happen so long as faction play and solaris remain as they.

[Nightbird]

R5D4, on 23 November 2020 - 01:26 PM, said:

I would argue it had an immediate dramatic impact followed on by a long term impact (both Solaris and Faction Play) because these game modes further leached away the population from a centralized source (quick play) and split them up into multiple buckets none of which interact together. This is largely why I don’t give the idea of fixing the matchmaker much credence given that PGI has made it SO much more difficult to pull together a proper team from the outset.

I think we're somewhat close in understanding each other so I'll try one more attempt. I'm not disagreeing that features and changes are important, people do come or go based on how PGI updates the game. To that end, the Green Retention line is never close to 100%, people that are happiest with the Matchmaker are still leaving for many different reasons.



The impact the Matchmaker has is the difference between the Green line and other lines, which are worse for every month of every year - on months when PGI does well, and on months when PGI releases a stinker. The question is, if the entire player base was in the Green line WLR bracket, how much better off would the player base and PGI be? The answer is we'd have a 50k pop instead of a 15k pop.

The other question is which features would you need to implement to have an equal impact to the Matchmaker? The answer is around every feature (except MM) that everyone has ever asked for, do that and you'll have a similar impact in improving retention. If we had an unlimited budget that would be possible...

R5D4, on 23 November 2020 - 01:26 PM, said:

I agree that improving the matchmaker is worthwhile endeavor as at least it might help stop/slow down the bleed of new players.

The Black line represents the most experienced players in the player base and they are leaving at the second highest rate. It's not just a new player problem.

[Rustyhammer]

Nightbird, on 19 November 2020 - 05:58 PM, said:

Since 50 out of 100 players already leave due to the Matchmaker, it is hard for another single feature to be more important wouldn't it? Therefore, the Matchmaker is king.

Did I convince you? Leave your Yes/No and Why in a comment below!

Hmm,
According to Steam numbers, only 51% of players completed the tutorial and only 40% played long enough to get their first victory: https://steamcommuni...00/achievements

You need to play at least 10 matches to appears on PGI stats. Assuming you can get 1 victory in 10 matches, 60 out of 100 players quit even before they finished 10 matches. Sorry but I cannot attribute their decision to the matchmaker, when 50% decided it's not even worth to play a match.

With only 24% of players managing to destroy 10 components, I highly doubt that 25% of players who completed the tutorial but decided to quit soon after did that because of the matchmaker and not because of horrible UI/performance issues/grind/etc...

[Nightbird]

Good point, the denominator is people who had 10 matches, but for a F2P game, some people installing to uninstall it isn't that strange. I don't see 10 games to start the game as a bad denominator, especially since we don't have anything else data-wise unless PGI decides to enlighten us. Best of all to be to have spending data, seeing what the paying customers do, but obviously we don't have that either.

[Rustyhammer]

It's not about the denominator.
How many matches needed to destroy 100 components? MWO is losing 10% population between 10 and 100 components achievements, and they are all pretty fresh accounts.
The new players tend to have low WLR due to inexperience and low population. The lower WLR graphs in your analysis is severely impacted by new player retention problem but you attribute it to the single issue - the matchmaker.

[Nightbird]

Those steam stats are cumulative, including days when the game first launched on Steam. You can't really read into it much.

I don't doubt there are newbie woes, but how much of it is Matchmaker related and how much of it isn't? Do you know?

How do you explain the high loss rate from the high WLR bracket comprised of experienced players?

[Rustyhammer]

Nightbird, on 23 November 2020 - 06:56 PM, said:

Those steam stats are cumulative, including days when the game first launched on Steam. You can't really read into it much.

Yes it's cumulative - just please don't say that 50 out of 100 players quit MWO due to the matchmaker when 60 out of 100 players who launched this game not even bothered to complete 10 matches.

Nightbird, on 23 November 2020 - 06:56 PM, said:

I don't doubt there are newbie woes, but how much of it is Matchmaker related and how much of it isn't? Do you know?

No idea - but I see new players posting many NPE issues and the matchmaker is just one of them. Do you think the reasons why all these Steam release players that left MWO after 1-2 month are fixed?

However I'm not the one saying the ideal matchmaker is the problem solver for MWO.

Nightbird, on 23 November 2020 - 06:56 PM, said:

How do you explain the high loss rate from the high WLR bracket comprised of experienced players?

Could be people reaching peak point, seeing no further goals to achieve in this game, getting bored and leaving? It may also be the matchmaker - but I can't say how much the decision to leave is impacted by the matchmaker comparing to i.e. lack of the end content for experienced players.

[Nightbird]

How many of those 60 players that didn't complete the tutorial never even installed MWO? It's a F2P game so some people just add it to their library.

I'm making a point that using that as a denominator doesn't mean much. 10 games at least indicates a willingness to give it a try, to learn. From that point of view, 10 games is not a bad choice, and it's also our only choice since our data is limited. You can criticize not having perfect data all you want, but that's a part of stats. You analyze and use what you have. If all you have are questions no one can answer, then how do we proceed? You either make choices using the data available (and PGI has more data than we do) or you ignore your data and base your decisions on hope and dreams.

Finally, the fact you automatically assumed low WLR are newbies and high WLR players are experienced means you agree the Matchmaker is broken. Someone who thinks the Matchmaker is fair (as one poster did) would claim few games as the cause of the outliers, not experience. The fact you automatically assumed the Matchmaker bias means intentionally or not, you're supporting my presentation that the Matchmaker is systematically failing at proving a fair learning environment for newbies, and a fun challenging environment for veterans.

[Rustyhammer]

I'm assuming we can project Steam stats to the whole MWO playerbase.
I'm also assuming new players have low WLR due to bad matchmaker and population issues.
---------------------------------------------------------------------------------------------------------------------------

I'm looking at it from achievement completion side:

10% of total players who completed the tutorial quit before playing 10 matches - these players are not even appearing in PGI statistics and excluded from your analysis. Removing 10% of data that most likely will fall within one category is a pretty big issue to me.

Another 25% of players quit before destroying 100 components, so no more than 1-2 month of play. Thanks to the bad matchmaker, most of them have low WLR - but do you agree it's a bit far fetched to say they left only because of the matchmaker?

I'm not debating that the bad matchmaker have negative impact on player retention.
I'm saying that lower WLR bracked data has other constant factors that cause excessive new player bleeding comparing to the established players with 1.0 WLR. Same for high WLR players - there are other constant factors for them to leave, not just the matchmaker. We are simply unable to see these factors due to limited data available to us. Proof is above: at least 10% of would be low WLR players left for other reasons than a bad MM. Eliminate these reasons and the low WLR graph will gets much close to the others.

I'm sure if we get a perfect MM with everyone having 1.0 WLR but nothing else changes, we will still see both new and skilled players leaving this game at accelerated rate comparing to average players who still have reasons to play.

Will the ideal matchmaker increase player's retention - absolutely. Is it the major factor in player retention - I doubt it.

[Nightbird]

Rustyhammer, on 24 November 2020 - 12:09 AM, said:

Proof is above: at least 10% of would be low WLR players left for other reasons than a bad MM. Eliminate these reasons and the low WLR graph will gets much close to the others.

You claim all of that 10% group that left before finishing 10 matches didn't do so because of the Matchmaker but some other reason. Based on what? 25%, 50%, 75% of them could have had 5 matches and said enough stomps, and quit, what's the support for your claim? I have no data.

Rustyhammer, on 24 November 2020 - 12:09 AM, said:

but do you agree it's a bit far fetched to say they left only because of the matchmaker?

there are other constant factors for them to leave, not just the matchmaker.

Your counter argument needs to prove that more than half of players left for reasons other than the Matchmaker. All you've done is twisting my claim to ALL the players left due to the Matchmaker, to make it easier to attack. This is called making a straw-man argument, don't do it.

The only systemic difference between the lines is that which is caused by the broken Matchmaker. Full Stop. There was no attempt to somehow put dissatisfied players into the Black and Red bracket lines. The differences between the lines, are therefore caused by the Matchmaker.

Edited by Nightbird, 24 November 2020 - 07:43 AM.

[MrMadguy]

The silliest proof, I've ever seen. All that proofs, that WLR-based MM is better, than skill-based one, are based on several wrongs assumptions, that automatically lead to several wrong conclusions. The biggest one - is that WLR is correlated well enough with skill. Don't forget, that at the end MM is about matching players with equal SKILL together. And WLR is 100% correlated with skill only in 1vs1 situation. More random players are matched with each other - more random factors are involved, that break this correlation.

This leads to the biggest problem with all that proofs - to assumption, that player with higher WLR will beat player with lower WLR, so WLR can be used as matchmaking rank in simulation. This makes all further simulation a priori flawed, i.e. make it lead to wrong conclusions. This makes such conclusions self-induced. I.e. assumption is used as proof of itself. I don't remember exact name of such logical fallacy.

Simple example. My WLR has always been around 1. But my quality of matches has always been low anyway. That caused me to quit. And, I guess, this problem can't be fixed purely by matchmaking. It's caused by low population.

For example we have perfect zero-based PSR now. I've been asking for it for a very long time. Yeah, my PSR drops. Yeah, it should mean, that quality of my matches should improve at some point and I should start to get compensation for all that bad matches. But no. It doesn't happen. Just because I've hit PSR floor already. I.e. no matter, how bad my matches will be and how low my PSR will drop - I will be matched with exactly the same players anyway. Simply because there are no more players in this game to be matched with. Devs should do, what players actually need and that would actually work. Not just waste their resources.

And I'm not 100% sure, but I guess, that only thing, that makes players keep playing any game - is new content. But problem is - this content should be good. Too much resources were wasted on "new" maps, like Forest Colony, River City, Terran Therma, Polar Highlands, etc. Nobody likes this maps, because they lack several major game mechanic features, any map should have. Mostly it's about routes to approach enemies.

Edited by MrMadguy, 24 November 2020 - 10:57 PM.

[GARION26]

Nightbird
Thanks for the in depth data dive. I greatly appreciate it.
However I think at best your data is hypothesis generating and not proof in support of your hypothesis of causes.

I agree with some of the critiques on your inferences on page 2 (Miss Greene, ShiverMeRivets etc.)

I think the best way to summarize my critique:
Is an extreme WLR the cause of loss of player drop out, or a predictor but NOT the cause?

As others have pointed out the extreme WLR groups are very different groups of players

Who is at the worst WLR - the newest players. New players try out a game and often drop out relatively quickly. A better WLR might keep them but in a FTP game new gamers come in and out of the population pretty frequently. If their WLR was 1:1 would they stay? Who knows. I know for FTP games I try out and don't like it doesn't matter if I'm 'winning' if it's not exciting me after a week or so it gets dumped.

Who has a ridiculously high WLR - in the world of MWO it's almost certainly people who've logged lots of hours - you can't just have great fast reflexes and a big bank account to master this game. There is a pretty steep learning curve (heat, weapon ranges etc) If someone gets to the the point the game is 'too easy' people lose interest.

Better way to do a data dive that might support your suggest analysis - follow people from account creation until termination (say no games in 3 months)
My hypotheses
1. I suspect WLR rises gradually from quite low to better
2. A large number of accounts drop within the first 2 months
3. Other accounts drop from time to time with minimal relationship between WLR outside of the most extreme ranges (for the reasons described above)

Note this critique doesn't disagree with your feeling that matchmaker could be better, and it's worth trying to make it better. It just says you can't use the data you have presented to make that conclusion.

[Nightbird]

MrMadguy, on 24 November 2020 - 10:01 PM, said:

Pretty much everything in your post is wrong, I don't know where to start. If you can provide evidence for anything you said, I'll give it a shot.

GARION26, on 25 November 2020 - 06:20 AM, said:

Who is at the worst WLR - the newest players.

As I posted above, the fact that the MM discriminates against new players proves it does not measure skill. If it measured skill, new players and veterans would all have around 1 WLR. Also the retention graph shows the impact this has on the player base. Sorry for the low effort post, see below.

----------------------------------------------------------------

Since PGI is being acquired, there's no point to discussing updates anymore. TL:DR Stats analysts like myself can find work in billion dollar companies because we find easy ways to improve earnings. This idea would have boost MWO's revenue 50-100% if implemented from the start. Crazy benefits for very little cost.

Edited by Nightbird, 25 November 2020 - 08:34 AM.

[GARION26]

No worries on the low effort post.

I'm quite familiar with the power of statistics - I am a health researcher where a large portion of my work is trying to make sense of large data sets. But what I do most is provide constructive guidance to other researchers about their work to help them do better quality work to answer the questions they are interested in. Take my own job experience as you will.

Again as I noted there is no reason not to argue for a 'better' matchmaker. I'm with you on that absolutely.

I do however argue the data you presented shouldn't definitive evidence that a better matchmaker would make someone trying out a game with a steep learning curve more likely to stick with it. Reasons laid out in my previous post and other's comments on page 2.

[Nightbird]

GARION26, on 25 November 2020 - 09:41 AM, said:

I do however argue the data you presented shouldn't definitive evidence that a better matchmaker would make someone trying out a game with a steep learning curve more likely to stick with it. Reasons laid out in my previous post and other's comments on page 2.

It is definitive evidence actually... I'm not trying to claim it will be successful with 100% success, but the difference between the lines show it would be successful to some degree.

[MrMadguy]

Nightbird, on 25 November 2020 - 08:16 AM, said:

Pretty much everything in your post is wrong, I don't know where to start. If you can provide evidence for anything you said, I'll give it a shot.

We should start from simple fact, that lower skill player has - lower his ability to affect results of match is. That's, what actually happens with me. In match with 2 pro players and 10 noobs on each team, it's that 2 pro players, who define result of match, while noobs are just passengers there. This makes noobs' WLR be affected by pros' WLR. And matchmaking rank should be personal rank. In any conditions. Not just in some perfect simulation conditions, where there are always enough players with exactly the same skill, players play 24/7, don't log out, don't have different time zones, etc. Faulty simulation proves nothing, you know.

[VonBruinwald]

MrMadguy, on 26 November 2020 - 07:55 AM, said:

We should start from simple fact, that lower skill player has - lower his ability to affect results of match is. That's, what actually happens with me. In match with 2 pro players and 10 noobs on each team, it's that 2 pro players, who define result of match, while noobs are just passengers there. This makes noobs' WLR be affected by pros' WLR. And matchmaking rank should be personal rank.

There's a flaw there, you are assuming good players will pull the teams performance up more than the bad players will drop it. With a functioning matchmaker the bad players will pull the team down an equal amount to what the good players raise it relative to the opposite team.

Edited by VonBruinwald, 26 November 2020 - 09:37 AM.

[Nightbird]

VonBruinwald, on 26 November 2020 - 09:27 AM, said:

There's a flaw there, you are assuming good players will pull the teams performance up more than the bad players will drop it. With a functioning matchmaker the bad players will pull the team down an equal amount to what the good players raise it relative to the opposite team.

Exactly, a bad match can be created from putting too many good players on one team, or putting too many bad players on the other. The two conditions are the same.

If you have a WLR<1, then it's because the Matchmaker thinks you're more skilled than you really are. If you have a WLR >1 then the Matchmaker thinks you're less skilled than you are. In either case, we need to give the Matchmaker prescription glasses.

Put another way, say you are a business owner with 5 customer reps whose job is it keep your customers happy. 1 is successful 80% of the time, 2 are successful 75% of the time, another one 50% of the time, the last 40% of the time. If you wanted to create a training on how to do the job best, which rep do you want to be the trainer?

Edited by Nightbird, 26 November 2020 - 10:16 AM.

[Kroete]

That all would be true, if we would not have groups and would have even weights in a match.
Until then all the talk doesnt matter much, as still having 5 tiers, even if we coulnt support tier 1 and 5 with enough players without opening the valves.

Edited by Kroete, 26 November 2020 - 11:01 AM.