63 replies to this topic

[Dr Cara Carcass]

MrMadguy, on 08 April 2018 - 02:28 AM, said:

My stats - are best proof, that WLR is completely meaningless for pure QP player.

Ok, your stats are horrible, and they jump a lot because you only play about 20 games per season.... But what do you want to show us here? Thats statistics need some time to get larger numbers?

The two seasons where you actually got more than 140 games are quite close in terms of win/loss. The month with 180 games had a few lights mixed in instead of 100% assaults+heavys - perhaps i found the reason why your win loss tanked a bit more.

Nothing is out of the normal here.

There is of course variance in win/loss even if the average matchscore k/d and what not are constant. But your own graph later showed that a good portion of WL is accounted for by matchscore alone. Now do that for k/d survive rate and so on. You might end up with a good explanation for all teh variable and their contribution to W/L.

Edited by Cara Carcass, 09 April 2018 - 03:38 AM.

[Dr Cara Carcass]

I want to add something.

Taxxian from 12DG wrote (better exanded on) an OCR tool for the jarls list. YOu amke a screenshot of the scoreboard before teh game starts, the tool reasds teh names and gives you the average team stats. After using it for a while i can predict the outcome of the match based on the numbers with above 90% accuracy.

I can tell you based on the matchscore relative to the other players in that game, who is in above their heads and will do less than 100, less than 200 and less than 300 dmg with an accuracy of 5 out of 6 players (we all know that every game has 5 players that totally underperformed each match). After a while you gain such a trust in the jarls list and its predictive powers, that you start to wonder why you have to play teh game at all.

The worst part is, that for Q-play swapping 2 Pilots the whole thing can be balanced. No need to find otehr palyers for a game. The same 24 pilots if sorted into two teams to balance the average matchscore of the teams will get you much closer games.

Sure you still can call out the guys that will underperformed, but both sides will have an equall amout of those pilots.
But a gamble on whose side will derp the most with the same amout of players derping is more interesting than seeing that you have the 6 worst potatoes on your team and basically knowing that at minute 4 in the game everything is decided because its already 0-6 since they ran in groups of two into a full team, with a 20 second delay between encounters. Bonus if you spot the guy with small lasers peaking at 700m who wodners why he didnt do any damage in a tier 1 match.....

https://github.com/h...coreboardHelper

The original link to the tool.

Edited by Cara Carcass, 09 April 2018 - 06:45 AM.

[Asym]

There are many good ideas posted and some nice logic being used.... But, the topic is dichotomy? Good and bad? And, if so, it must be mutally exclusive since we can't be part of both? Isn't the opposite of what MWO needs?

Personally, I'd remove all MM and use vehicle class and then balance by W/L as is.... That way, you carry a w/l based on that class and if you really are a terrible light driver, you'll end up with similar light drivers..... Otherwise, a universal W/L isn't nearly accurate for changes in weight class?

It's not an easy topic and I think Solaris is only going to make this worse.

[yrrot]

Solaris will at least give us more data (in theory) to look at. If we can access a similar level of data from Solaris matches and compare 1v1 performance and Jarl's list, we might start to see a pattern as to what gives the best indicator of "skill" in QP.

Definitely, playing a better mech will pad the stats for the average or better player. "Buy meta, boost stats" should really be PGI's mechpack sales tag, I guess.

Even if they don't go to a full blown Elo implementation for QP after Solaris, it would be nice if they carried over the per division approach to PSR or something. It would be nice to be able to play whatever fun-but-mediocre chassis without feeling like you are dragging the team down a bit.

[Nightbird]

W/L isn't a good stat to use for a good matchmaker, because a good MM will pull everyone's W/L ratio to 1. My suggestion is using match score and kills/match based on available stats. I explained the math in another thread but it's a pain to link on my phone.

[Spare Parts Bin]

The current PSR is no longer useful in my life. I play more FP than QP now. I understand FP game results do not affect your stats and achievements, is this true? If it is I care not for my Tier.

[Tarl Cabot]

Spare Parts Bin, on 09 April 2018 - 01:16 PM, said:

The current PSR is no longer useful in my life. I play more FP than QP now. I understand FP game results do not affect your stats and achievements, is this true? If it is I care not for my Tier.

FP stats do not have an effect on PSR... directly.... Indirectly though if you are doing well in FP on your builds, gameplay, teamwork, situational awareness and such (though many FP builds do not always pan out well for QP) that then can have a very possible effect in QP. The FP warning is for a new player to work things out in QP, obtain and level up mechs for FP but the reverse can be true too, if a player was to join a unit that does lots of FP but not alot of GP, that would be one way to flesh out one's skill and mechs.

Of course, the above is having an optimistic view on things...

Edited by Tarl Cabot, 09 April 2018 - 03:19 PM.

[Deathlike]

Asym, on 09 April 2018 - 12:20 PM, said:

There are many good ideas posted and some nice logic being used.... But, the topic is dichotomy? Good and bad? And, if so, it must be mutally exclusive since we can't be part of both? Isn't the opposite of what MWO needs?

Personally, I'd remove all MM and use vehicle class and then balance by W/L as is.... That way, you carry a w/l based on that class and if you really are a terrible light driver, you'll end up with similar light drivers..... Otherwise, a universal W/L isn't nearly accurate for changes in weight class?

It's not an easy topic and I think Solaris is only going to make this worse.

The problem is, as with many solutions... you don't have enough population to improve player selection. Forget group queue as it's not realistically possible anyways. There genuinely isn't enough people to do it w/o skewing with wait times. When you consider that when people play at off-peak NA times, the waiting can get real.

It doesn't work when the population does't actually improve their own skills either. Some try, but most won't grow. Part of that is a learning curve problem... the other is motivation (if you gave a damn, you would at least try to suck less yes?)

Edited by Deathlike, 09 April 2018 - 03:45 PM.

[SFC174]

Deathlike, on 09 April 2018 - 03:45 PM, said:

The problem is, as with many solutions... you don't have enough population to improve player selection. Forget group queue as it's not realistically possible anyways. There genuinely isn't enough people to do it w/o skewing with wait times. When you consider that when people play at off-peak NA times, the waiting can get real.

It doesn't work when the population does't actually improve their own skills either. Some try, but most won't grow. Part of that is a learning curve problem... the other is motivation (if you gave a damn, you would at least try to suck less yes?)

Population isn't really the issue. A proper matchmaker doesn't need to increase wait times much, if at all.

The key is that we can't expect that every player in the match will have approximately equal skill. This isn't really an issue since we don't get that now. What we should expect is that we will have well matched skill levels top to bottom across both teams. If we have a 10 level ranking system a reasonable match might have two top tier players (lvl 10) on each team, 3 lvl 9 and 7 lvl 8. Playing with/against better players is not bad matchmaking. In fact its the only way to truly get better if you're not top tier.

What I have a problem with is having the top 2 players in a match on one team and the worst 4-5 players on the other team. Take those same players and rearrange them as many have discussed using ranking (Jarl's list, etc) and the match gets more fair. It will never be perfect, but its the best we could ask for.

[Nightbird]

Deathlike, on 09 April 2018 - 03:45 PM, said:

The problem is, as with many solutions... you don't have enough population to improve player selection.

If I were to create a MM, it wouldn't do tiers at all. Every player gets a skill number based on their past performance. You can set the MM to launch the game when there's just 24 players waiting, and arrange both teams until their total skill is balanced.

If there's enough people, set MM to 48 people, when this number of people is reached, the top 24 gets put into one match, bottom 24 the other.

Repeat with 72, 96 etc, the more people you have before launching the games, narrower the skill band with each game between the top player and the bottom, and the longer the wait. But, in all cases the teams will be evenly skilled in total because the skill numbers are statistically computed to be proportional to a player's contribution to a team's win.

The number of people the MM waits for can be dynamic ofc, start with 120, drop to 96 after the first person in queue waits 60sec, 72 after 90sec, etc..

As long as I have data available, I can even weigh a player differently based on mech selected, loadout used, map selected, and mode selected. A skilled player using a 'fun' chassis and build can be dynamically scored lower than the same player using a meta chassis and build, not using a human guesstimate but a statistically modeled number.

Edited by Nightbird, 09 April 2018 - 07:56 PM.

[Asym]

Deathlike, on 09 April 2018 - 03:45 PM, said:

It doesn't work when the population does't actually improve their own skills either. Some try, but most won't grow. Part of that is a learning curve problem... the other is motivation (if you gave a damn, you would at least try to suck less yes?)

Again, once more unto the breach..... Improve skills? HOW??? What if you are an average players here fun the fun of it? Learning curve are directly proportional to population skills and game complexity... We have complexity in spades. Giving a darn has nothing to do with this.... IT IS A GAME.... We don't care how good you are? It isn't relevant... We will shoot you are fast as we shoot someone else. Suck less? Really??? You and many others take this game way too seriously....it's just a silly game where physics and science are replaced by make-believe and even then, PGI has dorked up even that made up science to the point that even the "experts" sometimes are at a loss....

No, the game ceased to be "fun" a long time ago... By the way, the average players were driven out of this game.... They wanted to stay but there was very little reason to do so: because it became so toxic it wasn't fun anymore.

[MrMadguy]

First thing - WLR and match score are correlated just because, surprise, victory gives you more MS, than loss, i.e. several bonuses, including for example flat 20MS victory bonus. Therefore higher WLR => higher AvgMS. Second thing - higher skill means more impact on result of match. What I constantly feel - is that I have zero impact on result of match due to my low skill and therefore WLR is completely meaningless stat for me. Outcome of any match would be exactly the same, if I would simply go AFK. I don't want to say, that WLR is bad - I just try to say, that if they're correlated anyway, then using both WLR and AvgMS to measure skill - is really good idea, as it gives much more accuracy, especially for low end players.

[Zergling]

MrMadguy, on 09 April 2018 - 09:00 PM, said:

First thing - WLR and match score are correlated just because, surprise, victory gives you more MS, than loss, i.e. several bonuses, including for example flat 20MS victory bonus. Therefore higher WLR => higher AvgMS.

There is too great a difference in Average Match Score due to W/L than is possible due to the winning bonus.

Eg, a 2.00 W/L is only winning approximately 33% more battles than a 1.00 W/L player, but averages approximately 330 Match Score versus 200.

I mean, 1.00 W/L over 200 battles is 100 wins versus 100 losses. A 2.00 W/L over the same 200 battles is 133 wins to 67 losses.
And 3.00 W/L over 200 battles is 150 wins versus 50 losses, only 50% more wins than a 1.00 W/L, yet the average Match Score for 3.0 W/L is approximately 430.

That is only 33% more wins for 130 more average Match Score, and 50% more wins for 230 more average Match Score; the winning MS bonus cannot account for such differences.


Like, the winning bonus definitely would causing some of the difference in Average Match Score, but it would be fairly minor, because there just isn't enough of a difference in wins.
Eg, the amount caused by a 20 bonus to MS per win would only be 10 points more average MS for a 3.00 W/L player versus a 1.00 W/L player, leaving the other 220 points of average MS being due to greater player performance.

MrMadguy, on 09 April 2018 - 09:00 PM, said:

Second thing - higher skill means more impact on result of match. What I constantly feel - is that I have zero impact on result of match due to my low skill and therefore WLR is completely meaningless stat for me. Outcome of any match would be exactly the same, if I would simply go AFK.

Players definitely can have an impact on your match with low skill; a negative impact, much as an AFKer or suicider would have.

MrMadguy, on 09 April 2018 - 09:00 PM, said:

I don't want to say, that WLR is bad - I just try to say, that if they're correlated anyway, then using both WLR and AvgMS to measure skill - is really good idea, as it gives much more accuracy, especially for low end players.

Few people want to look at a single statistic as a performance measurement, as highly misleading results occur when this is done.

I prefer to look at four statistics at the same time: Wins/Losses, Kills/Deaths, Kills/Battles and Average Match Score.

Edited by Zergling, 09 April 2018 - 09:29 PM.

[MrMadguy]

Zergling, on 09 April 2018 - 09:23 PM, said:

I prefer to look at four statistics at the same time: Wins/Losses, Kills/Deaths, Kills/Battles and Average Match Score.

Kills are accounted in match score. WLR and AvgMS are enough. But sole WLR isn't.

Hard to explain. From WLR's point of view filling team with random players with random skills, whose average skill is 5, is equal to filling teams with players, whose skill is exactly 5. It's simple math: 2 + 2 = 1 + 3 = 0 + 4. Average is always 2. Larger teams are - smaller difference is and bigger chance, that averages will be equal. So, law of large numbers stops working here, as you're not the only random variable, that affects result of match - there are another 23 random values, that completely obscure your effort toward victory. And real situation is much more complex, as skill - isn't only random variable here. There other random variables, such as time frames, players are available at, 'Mechs and builds, they choose to play at certain moments of time, teamwork, etc. But MM should also guarantee 1 vs 1 balance. And this can only be achieved via taking personal performance into account. And WLR based MM would simple feel lost among such large bunch of random variables, affecting outcome of any match.

May be I'll make some sort of simulation to prove it. I'll think about it.

Edited by MrMadguy, 09 April 2018 - 10:46 PM.

[Zergling]

MrMadguy, on 09 April 2018 - 10:34 PM, said:

Kills are accounted in match score. WLR and AvgMS are enough. But sole WLR isn't.

It is possible to inflate Average MS without scoring much kills, like LRM boating.

MrMadguy, on 09 April 2018 - 10:34 PM, said:

Hard to explain. From WLR's point of view filling team with random players with random skills, whose average skill is 5, is equal to filling teams with players, whose skill is exactly 5. It's simple math: 2 + 2 = 1 + 3 = 0 + 4. Average is always 2. Larger teams are - smaller difference is and bigger chance, that averages will be equal. So, law of large numbers stops working here, as you're not the only random variable, that affects result of match - there are another 23 random values, that completely obscure your effort toward victory. And real situation is much more complex, as skill - isn't only random variable here. There other random variables, such as time frames, players are available at, 'Mechs and builds, they choose to play at certain moments of time, teamwork, etc. But MM should also guarantee 1 vs 1 balance. And this can only be achieved via taking personal performance into account. And WLR based MM would simple feel lost among such large bunch of random variables, affecting outcome of any match.

You're overthinking it.

When teams are random, the only influence on W/L over a large number of battles is personal performance, because nobody will end up with better teammates than anyone else.

All bigger teams does is compress winrates closer to 1.00, because it is harder for high skill players to carry a team, and bad players are less likely to screw their team over.

[Nightbird]

Hoping against hope that PGI is reading this type of threads

[East Indy]

There's a side debate going on over whether win-loss is a magical indicator of performance potential. It isn't. There's this poor guy I see every now and then who's always in an assault. My W/L rate is only 17% more than him. My K/D is 200% (3 times) higher, and my average adjusted matchscore is 60% higher.

What's missing from the analysis is the effect of guys like him who, in numbers, pull down teams of guys who at least pull their weight.

There's a difference between a cascade and a roll.

Edited by East Indy, 10 April 2018 - 03:28 PM.

[yrrot]

Random thought: is QP solo building one team at a time? I've seen that mentioned somewhere by other folks here, and it doesn't make sense from an initial development of a solo drop system. It *does* however make a ton of sense if it's the product of splitting out the solo and party queues. Group queue building a 12 man team from random groups (and previously including solo players as fill), then matching two 12 mans seems like what this MM would be doing in the background--especially given legacy implementations/code.

If that's the case, then the "just swap players from one team to another for balance" isn't as simple. It doesn't leverage the legacy code, and requires new developments that likely are risky to the existing MM code.

[yrrot]

As for the WLR discussion: while it is true that a large sample set should mitigate the randomly bad teammates from dragging down one's personal WLR more than anyone else's, how large of a set is that? I'm not a statistics expert, but the law of large numbers isn't talking about ~1000, it's talking about 100's of 1000's. That being said, if the leaderboard data separated out solo and group queue data, we might see the solo queue data reflect that effect, even in the relatively small sample size (of one player's games). With the two being joined, it makes WLR a less reliable stat.

Basically, larger groups of players inherently have better organization and teamwork, and often higher average skill. Grouping up as such, inflates WLR because their combined advantage will outweigh the skill of most of the other groups they match against. Even well organized mediocrity can overcome a few higher skilled players and mop up the rest, essentially. Since there seems to be a lot of semi-casual, small groups in queue, I think larger teams might often run into less organized groups more often than they run into other large, organized groups. Since we can't split out group play from the WLR, it loses some clarity.

Average match score has some glaring problems, but at least it gives a rough indication of a player's contribution. The obvious problem being that damage and killing blows do not equate directly to talent. Highly accurate folks using PPFLD weapons might do less, but more focused damage, for example. Someone less accurate with a splash weapon (LBX, MRM, LRM, etc) may put up high numbers and get killing blows, but not really contribute much to the actual CT kill by shaving armor off of all of the other components that didn't matter. There's also several things that don't contribute to match score nearly enough, like capping objectives. I've seen games won entirely because of a pair of lights surviving and capping, only to have the lowest match score in the game. Even though they were playing to the objectives more than anyone else in the game.

Really, if we want to figure out the best metric to use for skill, we need more granular data than we have now. Separating solo/group queue, adding FP data, and sorting out things like KMDD, Solo Kill, Assists, spotting, etc. are all things we would want to look at and compare. And that's before we even talk about weight class and variants.

TL;DR: We're bickering about which stat to use, when none of the data we have is clean enough in the first place.

[Deathlike]

SFC174, on 09 April 2018 - 07:38 PM, said:

Population isn't really the issue. A proper matchmaker doesn't need to increase wait times much, if at all.

The key is that we can't expect that every player in the match will have approximately equal skill. This isn't really an issue since we don't get that now. What we should expect is that we will have well matched skill levels top to bottom across both teams. If we have a 10 level ranking system a reasonable match might have two top tier players (lvl 10) on each team, 3 lvl 9 and 7 lvl 8. Playing with/against better players is not bad matchmaking. In fact its the only way to truly get better if you're not top tier.

What I have a problem with is having the top 2 players in a match on one team and the worst 4-5 players on the other team. Take those same players and rearrange them as many have discussed using ranking (Jarl's list, etc) and the match gets more fair. It will never be perfect, but its the best we could ask for.

Population is always an issue. Compared to other games, we're building at least double the average team size compared to League of Legends or Overwatch, while not even having half the population or the skill ranges needed to complete such a task.

What happens often is what normally happens... a "valve" opens, where the game eventually ignores restrictions (whether it be tonnage or skill or a combination) and just builds the match. So it's not really succeeding, it's just failing casually.

Nightbird, on 09 April 2018 - 07:45 PM, said:

If I were to create a MM, it wouldn't do tiers at all. Every player gets a skill number based on their past performance. You can set the MM to launch the game when there's just 24 players waiting, and arrange both teams until their total skill is balanced.

If there's enough people, set MM to 48 people, when this number of people is reached, the top 24 gets put into one match, bottom 24 the other.

Repeat with 72, 96 etc, the more people you have before launching the games, narrower the skill band with each game between the top player and the bottom, and the longer the wait. But, in all cases the teams will be evenly skilled in total because the skill numbers are statistically computed to be proportional to a player's contribution to a team's win.

The number of people the MM waits for can be dynamic ofc, start with 120, drop to 96 after the first person in queue waits 60sec, 72 after 90sec, etc..

As long as I have data available, I can even weigh a player differently based on mech selected, loadout used, map selected, and mode selected. A skilled player using a 'fun' chassis and build can be dynamically scored lower than the same player using a meta chassis and build, not using a human guesstimate but a statistically modeled number.

This would've been something that should've implemented or discussed thoroughly when PGI was messing with the MM. At this point the skill needed to accomplish that is closer to Lostech (the last known decent programmer they had in Karl left for Amazon after all).

Asym, on 09 April 2018 - 08:05 PM, said:

Again, once more unto the breach..... Improve skills? HOW??? What if you are an average players here fun the fun of it? Learning curve are directly proportional to population skills and game complexity... We have complexity in spades. Giving a darn has nothing to do with this.... IT IS A GAME.... We don't care how good you are? It isn't relevant... We will shoot you are fast as we shoot someone else. Suck less? Really??? You and many others take this game way too seriously....it's just a silly game where physics and science are replaced by make-believe and even then, PGI has dorked up even that made up science to the point that even the "experts" sometimes are at a loss....

No, the game ceased to be "fun" a long time ago... By the way, the average players were driven out of this game.... They wanted to stay but there was very little reason to do so: because it became so toxic it wasn't fun anymore.

No. Elite players have eroded/disappeared on their own. The bulk of players are really just players that don't care about their stats and/or just willing to hand over fist about money for nostalgia, combined with people that don't quite know any better.

This is why gameplay is dramatically different during events. They play to "figure it out", but struggle in the mechanics of success. One could easily argue that the playerbase is what PGI wanted... to make the most money to survive. This is the consequence of their actions.