76 replies to this topic

[Nightbird]

I think something that every businessman has wished for at one point is a crystal ball. Which direction, decision will lead to a successful and profitable future? Most of the time one can only make a guess, intuit the best course, and hope for the best. Even then, one never really knows if the final choice was the best one.

Something rare happened in MWO of all places a while ago. There was an opportunity to not only make a change in direction, but the choice had predictable consequences. MWO wanted to update its Matchmaker to give its players a better match experience, and opened up to the community for proposals. Unlike adding new features which may or may not be well received, making matches is a math problem, and math problems can be solved.

So that is what I did. I took the Matchmaker solution that PGI selected, alongside my own proposal at the time, and crunched the numbers to see what the results will be months later. I then posted these results for all to see, 9 months ago, before the update even happened.

The predictions were as below. The most easy metric to compare against is the sum of the 3 center bars.



We haven't quite reached 480 average games in the last predicted graph because the increase in pop from COVID slowed down the progression of average games (though an increase in pop is always a good thing), but here are the results up to the end of the March 2021 season:



So the predictions and actual results are close, so what?

Well it turns out that how happy players are, and their willingness to keep playing the game (their retention rate) tracks closely with their WLR.




People close to 1 WLR represented by the green line are consistently the happiest season to season. Players that get further away from 1 WLR in both directions are less happy. This seems to indicate that new players are being given games which are too difficult for them, and they leave, and veteran players are given games which are too easy for them, and they leave as well.

Our goal is increasing player happiness, which means better retention and a larger player base. As reference, we can see the impacts of past decisions in the retention graph.




In (A) we see the baseline retention rate when mech packs were still released monthly. In (B ) we see what happened when MWO was announced to be on maintenance mode with no new development. In (C) we have Daeron being hired, and MWO coming off maintenance mode with new balance and content updates promised and delivered.

The impact of business directions (A), (B ), and (C) is clear. At the same time, player happiness is still consistently being affected by the Matchmaker. If we wanted to improve retention rates and grow MWO's player population further, wouldn't we want to increase the proportion of the player base in the green 1 WLR ratio line? Currently as of the last season 28% of the population is in the green line, if we can boost it to 100%, then this action can boost the average retention rate from (C) to (D), an even bigger improvement than from (B ) to (C).

But is boosting the retention rate to (D) even possible? Coming back full circle to the start of this post, the Matchmaker is just math and the results of math is predictable. The distribution of WLR from the current Matchmaker was accurately predicted 9 months ago. The Matchmaker I proposed then would have put 75% of the population into the green line. With more work, a Matchmaker that does so with 90-95% of the population can be created.

Lastly, how big an impact are we talking about here? A "dumb" way to do it is to look at steam charts (https://steamcharts.com/app/342200), and add 10% to the monthly change every month. -10% becomes 0%, 0% become +10%, +10% becomes +20% etc. There is a better way to calculate this of course, but I think this post has become long enough, don't you?

Edited by Nightbird, 10 April 2021 - 01:02 PM.

[Squishy- Iv4]

I find this data to be heavily dependent on several factors outside of WLR and retention. Happiness and player satisfaction are completely irrelevant from performance as they are abstract and unquantifiable. However I do appreciate the extensive research as well as time dedicated towards such a proposal. I must however point out that you do have a point to player retention to a certain extent reflecting player happiness.

For the most part I disagree with this, but I completely appreciate this topic being discussed for it is intriguing and rather important. I will upvote this.

[PocketYoda]

Math=/=Reality..

No wonder the game industry is like it is..

[Heavy Money]

This analysis is full of assumptions, but your conclusion that improving the matchmaker will improve the game is almost certainly true.

I'd question if people who are above 1:1 W:L are quitting from boredom due to winning too much, as opposed to boredom from stagnation of the game in other ways. I have 1:1 WL and I'd certainly like to win more!

I don't think its a stretch to say that the vast majority of players want to have a good battle, and then win. Sure you'll get bored if you stomp everyone all the time and there's no challenge, but due to the nature of the game, only the top few % of players are going to win that reliably. Its much more plausible those above 1:1 WL get bored from lack of new content, balance issues, etc. Its the good battles that matter. You don't mind a loss if the fight was fun. And nobody ever said "Gee i love the battles, but my high win rate means I gotta move on now!"

So W:L as a data point is only relevant as a proxy for having a fun battle. So we need to be careful. There's lots of ways we can "solve" for the problem of WL that won't make battles more fun!

Overall though, I don't think there's any doubt that improving match quality will improve retention of people with less than 1:1 WL. So it should be done. The issue is what exactly is match quality? What makes a match fun, and how do we make a system that selects for it?

We can start with some basic assumptions, like that making matches more fair in terms of tonnage and player skill will make them higher quality. But will it? Many of my most memorable matches were memorable because of a tonnage imbalance, or because I fought a team stacked way against me. Normalizing things too much might just make everything the same all the time, and kill the fun. Its not a simple problem.

And of course, if getting a "higher quality" match means I can't play with my buddies, then what's the point?

Edited by Heavy Money, 10 April 2021 - 06:07 PM.

[Nightbird]

There are parts of my post that is data, and parts that is my interpretation. The part where the player group with the highest WLR are leaving at the second worst rate is data, I did not make that up. If people feel my interpretation - that they are insufficiently challenged by the Matchmaker - is wrong, feel free to add your own interpretation.

For example, one interpretation that was shared with me is that veteran players are bored with the stagnant meta, the agility and weapon nerfs, the lack of new content. This is IMO a fair interpretation, and one we can put to the test. This month's patch will host an extensive weapon re-balancing effort, and over the next half year we'll see updates to mech agility, quirks, re-sizing, new mech packs, map updates, and more. If over the next half year the retention rate of veteran players improve, becoming better than the 1 WLR players, then this interpretation holds true. Like my prediction 9 months ago, this interpretation is one where we have to revisit in 6 months' time to see if the evidence then confirms it or not.

[ScrapIron Prime]

Samial, on 10 April 2021 - 05:37 PM, said:

Math=/=Reality..

No wonder the game industry is like it is..

Statistics don’t equal reality either, but it heavily relies on it. And this is a good analysis. You can draw different conclusions from raw data, but you need raw data like this to have an informed discussion. Thanks Nightbird!

[w0qj]

Do appreciate your hard work in producing these data sets & charts!

Just like in economics, the data charts is just a snapshot of what had already happened.
If one is to try predicting the future by extrapolating from the charts, it's like trying to drive a car forward while looking only at the rear view mirror (ie: it does not work). This comment is from critique of economic predictions by the way, especially for predicting inflation.

MWO is in a crucial state this year (2021)--with a big influx of returning MWO players and some new players, it is critical that MWO retains many/most of these to increase its player base. More players statistically means more sales; online game sales volume just a game of percentages.

I personally believe that Matt & Daeron made the correct and bold move to keep up this new player momentum, to dive headfirst with a radical ~April 20th PTS/Gulag/Cauldron weapons/equipment pass update, and then make iteration/adjustments regularly until things settle down, while listening to player feedback.
Here's to hoping that the April 20th patch weapons/equipment changes would be well received!

- - - - - - - - - - - -
Perhaps PGI should consider cross-product marketing also.
How about freely bundling all MW5 DLC purchases with MechWarrior_Online Special Variant Mechs, to encourage MW5 players to play MWO also? Oh, and this is retroactive, even for customers who have already purchased DLC from Epic.
This would maximize MWO player footprint and hopefully new player retention numbers.

Edited by w0qj, 10 April 2021 - 10:13 PM.

[Antares102]

Uh wow kudos for this analysis man!

[Gagis]

Nightbird, on 10 April 2021 - 08:45 PM, said:

The part where the player group with the highest WLR are leaving at the second worst rate is data, I did not make that up. If people feel my interpretation - that they are insufficiently challenged by the Matchmaker - is wrong, feel free to add your own interpretation.

For example, one interpretation that was shared with me is that veteran players are bored with the stagnant meta, the agility and weapon nerfs, the lack of new content. This is IMO a fair interpretation, and one we can put to the test.

I believe both of these intrepretations are true, but there's also a third part.

MWO has no End Game. Once a player is good enough that Quick Play provides no meaningful feeling of growth and archievement, there is nowhere to go except competitive play. Since not all players want to commit to scheduled practice and match times, most players quit after getting good.

PGI will eventually have to recreate Faction Play or think of a new alternative kind of End Game, or the game will die. Matchmaker and balance are more immediate concerns and ones that can be dealt with with scarce funds, but eventually we'll run into the lack of end game as a crippling problem. It's gonna be a tough nut to crack.

[Bud Crue]

Is there a way to factor in nostalgia in regard to retention? Most of those I play with are very long term players indeed, with my having actively played "only" since 2015 being a definite outlier. Pretty much the only reason we play is due to the nostalgia for the IP and because the game has become a social habit. The players in question are mostly represented by the green line (W/L ~1) but some are extreme red liners and a couple are black liners. So it seems to me that there is potentially a significant number of players who aren't going anywhere, regardless of what PGI does to the game or how we perceive our own performance.

On a sort of related note, your analysis makes me wonder how the retention rate numbers over time would compare to player spending. There would presumably be a big spike before the game even started via the founders cash, but after that I suspect spending would track pretty close to retention rates. Then again, maybe not. Most of the long term playing folks that I play with haven't spent a dime on the game in years, regardless of W/L ratio. Of those that do spend, some do so freely and without consideration of PGI's conduct or changes to the game. Another thng, just before the game went into maintenance mode Russ mentioned a couple times how, despite the age of the game, 100,000 people were still logging in every month, and I always thought "yeah but how many are spending anything once they log in?" Apparently not many (or not enough in any case), since a couple months later when maintenance mode was announced, he said that mech packs weren't profitable enough and hadn't been for some time (paraphrasing). So maybe spending patterns are wholly different than the retention numbers.

Sorry, just rambling. Your post was very thought provoking.

[Nightbird]

Gagis, on 11 April 2021 - 01:57 AM, said:

Once a player is good enough that Quick Play provides no meaningful feeling of growth and archievement, there is nowhere to go except competitive play. Since not all players want to commit to scheduled practice and match times, most players quit after getting good.

I see this as another way of wording what I said about the Matchmaker giving too easy matches. Our population is capable of giving more challenging matches, which can extends the feeling of growth and achievement, just as it is capable of giving a more nurturing environment for beginners to learn. A Matchmaker that gives more fair matches to beginners will create more 1 WLR players. A Matchmaker that give more challenging matches to veteran players also create more 1 WLR players. When I say 1 though, I don't mean =1. The more and less skilled you are, the further you will be from 1. Just not as far as today.

Gagis, on 11 April 2021 - 01:57 AM, said:

PGI will eventually have to recreate Faction Play or think of a new alternative kind of End Game, or the game will die. Matchmaker and balance are more immediate concerns and ones that can be dealt with with scarce funds, but eventually we'll run into the lack of end game as a crippling problem. It's gonna be a tough nut to crack.

I didn't bring this up, but FP is the reason for the first big dip in the retention graph. Its effect is more on the macro side, and less on the veterans only side.

[Nightbird]

w0qj, on 10 April 2021 - 10:11 PM, said:

If one is to try predicting the future by extrapolating from the charts, it's like trying to drive a car forward while looking only at the rear view mirror (ie: it does not work). This comment is from critique of economic predictions by the way, especially for predicting inflation.

"Those that fail to learn from history, are doomed to repeat it."

Bud Crue, on 11 April 2021 - 05:02 AM, said:

snip

There are a lot of things that PGI could do but I couldn't for lack of data. For example, they can look at the people that play more FP than QP and see if their spending is more or less than those that play more QP than FP. This information on preferred mode, and of course spending data, is not available but would be simple to graph.

Edited by Nightbird, 11 April 2021 - 06:16 AM.

[Flyby215]

Nightbird, on 11 April 2021 - 06:11 AM, said:

There are a lot of things that PGI could do but I couldn't for lack of data. For example, they can look at the people that play more FP than QP and see if their spending is more or less than those that play more QP than FP. This information on preferred mode, and of course spending data, is not available but would be simple to graph.

I had assumed it was a cold, calculated decision. More money from group players than solo players; so that is who must be catered to. As part of said calculation, I am no longer the target player, I stopped spending. Not to be spiteful or vengeful, just math in business

What is surprising to me though; and please correct me if I am wrong here; from what I understand is that PGI will receive financial bonuses from their new owner if they meet certain revenue/profit targets. To that end, I am amazed PGI hasn't pulled all the stops from PTW ammo, making Premium time in some way mandatory; or more sinisterly: secretly downgrading players into easier Tiers when they play hero mechs to make pay-mechs appear more appealing (I can confirm this isn't happening given how crappy this month has been!).

Edit: Added text for clarity.

Edited by Flyby215, 11 April 2021 - 12:04 PM.

[Escef]

MechaGnome, on 10 April 2021 - 05:37 PM, said:

Math=/=Reality..

No wonder the game industry is like it is..

Numbers don't lie. But nor do they tell the whole story.

Done properly, the numbers reflect reality. Done improperly, they reflect bias.

It is also possible to have improperly done your numbers and stumble into the correct answer anyway.

As for what the OP has to say on it, I'll leave you to make your own decision on that.

[DUMAR]

Escef, on 11 April 2021 - 12:13 PM, said:

Numbers don't lie. But nor do they tell the whole story.

Done properly, the numbers reflect reality. Done improperly, they reflect bias.

It is also possible to have improperly done your numbers and stumble into the correct answer anyway.

As for what the OP has to say on it, I'll leave you to make your own decision on that.

[Nightbird]

Flyby215, on 11 April 2021 - 12:02 PM, said:

I had assumed it was a cold, calculated decision. More money from group players than solo players; so that is who must be catered to. As part of said calculation, I am no longer the target player, I stopped spending. Not to be spiteful or vengeful, just math in business

If PGI had done the math I'd be relieved. My guess would be on the FP players being the larger part of the founders and paying good money for a BT multiplayer game. When they left, PGI's bottom line probably got a big shock. We'll never know.

[PocketYoda]

Escef, on 11 April 2021 - 12:13 PM, said:

Numbers don't lie. But nor do they tell the whole story.

Done properly, the numbers reflect reality. Done improperly, they reflect bias.

It is also possible to have improperly done your numbers and stumble into the correct answer anyway.

As for what the OP has to say on it, I'll leave you to make your own decision on that.

Yes they do lie.. And companies can do their own meta data.. it still fails miserably most of the time, also half of the data can be faked and regularly is with games companies..

Edited by MechaGnome, 11 April 2021 - 10:32 PM.

[Nightbird]

MechaGnome, on 11 April 2021 - 10:30 PM, said:

Yes they do lie.. And companies can do their own meta data.. it still fails miserably most of the time, also half of the data can be faked and regularly is with games companies..

"Most of the time", "Half the data can be faked"? Yes, stats shared without source or methodology is 100% fake. Good stats explain where it comes from and how it is calculated so that others can reproduce and verify the information, and if they can't be reproduced or the methodology is shaky, become the grounds to criticize it. Fake stats don't include these because they do not have a good source, or the methodology is "What I say is always true, even if I just pulled it out of my butt"

Edited by Nightbird, 12 April 2021 - 07:39 AM.

[Vlad Ward]

Hey NB. Cool analysis. Were any filters applied to the population, eg >X total games played?

Edited by Vlad Ward, 12 April 2021 - 02:37 PM.

[Nightbird]

Vlad Ward, on 12 April 2021 - 02:37 PM, said:

Hey NB. Cool analysis. Were any filters applied to the population, eg >X total games played?

Good question. There is a >= 20 games played filter for WLR graphs because someone pointed out only having 10 games would be too random. This is cumulative for the seasons indicated, not just per season. There was no filter used for the retention graphs though because we want to know if they stayed or not.