152 replies to this topic

[Ex Atlas Overlord]

Orville Righteous, on 12 July 2016 - 06:29 AM, said:

VictoriaSeymore is a good argument for adding a Tier 0. I wasn't surprised to him ranked near the top when the leaderboards came out.

Seriously, these stats are TOO good.

Like... I'm going to need a live video of every single match back to back, using pre-tested hardware, with a drug screen, a letter of recommendation from his/her mom, and somehow not connected to the internet.

The KDR, I understand. Get a few kills a game, and then hide to save KDR for the rest of the match. He might just be REALLY good at hiding...

but the W/L..I'm in full flat earther mode...I just refuse to believe someone could play THAT many games.....and manage to avoid the derptastical pug induced loss THAT much.

There is no way that's pure, random, solo Q... has to be sync dropping or group Q in there.

Edited by Ex Atlas Overlord, 12 July 2016 - 04:46 PM.

[VictoriaSeymore]

If I perform consistently well, I should be able to carry my teams. Now there are times where my team-mates are completely useless and we lose. However, in the majority of games, if I play to my standards and don't make too many mistakes, I can solo carry the other 11 players. Here's an old interview which details some of my thoughts towards solo queue, matchmaker, and carrying. http://www.qqmercs.c...iew-kaffeangst/ It's from a few years ago, but it still remains the same. Now... PGI should have a better method of match-making than to throw together teams that require one or two players to balance the inferior members of their side.

It's 100% Solo Queue. One cannot maintain an average of 640 match score and close to 1100 damage while relying on a group to boost their Win/Loss. Too much sharing of damage and kills to have the best numbers this way.

Edited by VictoriaSeymore, 12 July 2016 - 05:02 PM.

[1453 R]

Gotta say...I read Blastman's OP for this thread and almost literally threw up in my mouth.

Do you people ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, ever, EVER EVER EVER



And think to yourselves...:

"Hey...Piranha's revised the matchmaking system, like, four hundred and seventy-three different distinct times now and yet we never get to this perfect Utopia of infinite easy nail-biter 12-11 matches everyone keeps b!tch!ng at Piranha over. This is sort of a nichey game with a not-enormous playerbase, and it's also the sort of game where snowballing happens at the drop of a hat simply because of the proper and righteous lack of respawns and no ability to 'heal' mid-match.

Not to mention that even T1 Ubergods with awesome stats, years of experience, and rakish good looks can decide to get drunk one night before jumping into the game and figure driving their old mono-ERLL Mist Lynx sniper would be fun, or that the T2 'chronicbad' garbagelord I spend all of my forum time deeply insulting and trying to get to uninstall might get lucky, or find his zone, and tear folks up the matchmaker wasn't expecting him to! Heck, one lucky alpha to the right opponent early in a match can butterfly-effect the whole thing! I mean hell, how in the world do I expect an algorithm-based matchmaking system to possibly account for the infinite variation and unpredictability of human idiocy?!

Seriously, every other multiplayer game released in the last ten years has been subject to the exact same accusations of 'useless matchmaking turds!' by their communities - I gotta wonder if this is an endemic problem in the fundamental nature of online multiplayer games, and not just Piranha being bad at game dev-ing the way I keep trying to convince the whole world they are."

Edited by 1453 R, 12 July 2016 - 05:12 PM.

[Hit the Deck]

Ex Atlas Overlord, on 12 July 2016 - 04:37 PM, said:

...
but the W/L..I'm in full flat earther mode...I just refuse to believe someone could play THAT many games.....and manage to avoid the derptastical pug induced loss THAT much.

There is no way that's pure, random, solo Q... has to be sync dropping or group Q in there.

Not speaking about Victoria/kaffee but according to my observation in the QP, good players who always bring optimized 'Mechs and don't play around do carry and bring his/her team to victory most of the time (in QP).

The key to their consistency is bringing optimized (meta) 'Mechs and not changing 'Mechs so often from game to game.

[Hit the Deck]

VictoriaSeymore, on 12 July 2016 - 05:01 PM, said:

...Here's an old interview which details some of my thoughts towards solo queue, matchmaker, and carrying. http://www.qqmercs.c...iew-kaffeangst/

.
.

VictoriaSeymore said:

Q: What do you do when you’re just playing for fun?

I don’t have fun playing MWO. It’s serious business. I play to win every match, and require the same of my lance-mates. If we’re winning, I’m happy, so I guess that’s fun enough for me.

LOLZ

[The Rogue Wolf]

Dryderian, on 12 July 2016 - 02:27 PM, said:

Overall the groups seem very uneven and a good fight does not happen very often, you stomp a team(very rarely) and get stomped a lot. I really thought the fights get more tactical, planned and coordinated when I get to higher tiers.

I found that when I hit Tier 3, the players got a lot more skillful, coordinated and cohesive.

The players on the other team, that is.

(I kid, I kid. But Tier 3 seems to be the midpoint between "how do I shot lazur" and "I actually pilot a mech in real life", where many players think they know what they're doing, but still have more to learn.)

[adamts01]

Littlerift, on 12 July 2016 - 05:35 AM, said:

We don't want 100% perfect skill-based matchmaking because under such a system eventually everybody ends up with a 50% win rate and a K/D of 1.

A 50% W/L and 1:1 K/D would be awesome. I'd take 10 great losing fights over 10 winning stomps any day or the week.

Percimes, on 12 July 2016 - 10:22 AM, said:

Sorry to say this but the stats are almost worthless, just barely worth more than the MM.

KDR and damage can be incredibly misleading, but W/L isn't. A good W/L means you consistently do what your team needs you to do.

Ex Atlas Overlord, on 12 July 2016 - 04:31 PM, said:

Then perhaps in order to prevent needless circular point avoidence for the sake of semantical sidetracks we should replace the word "experience" with " experience while actually giving a **** "

Any general argument I make refers to an average player. Finding some oddball example like you did here "I highly doubt anyone that has never played MWO will somehow become a worse player the longer they spend playing...' is what's counterproductive.

Ex Atlas Overlord, on 12 July 2016 - 04:37 PM, said:

but the W/L..I'm in full flat earther mode...I just refuse to believe someone could play THAT many games.....and manage to avoid the derptastical pug induced loss THAT much.

There is no way that's pure, random, solo Q... has to be sync dropping or group Q in there.

There are players that good. When I played my Cheesy Timby I had a 5:1 kdr and a 3.5:W/L. The gameplay sucked, I'd use my pugs for bait, win most matches, but it was lame as ****. I'm much happier now running around in my light, doing my part, and accepting I'm at the mercy of MM. But some people only care about the win, run whatever FOTM OP build is, and need to see their name at the top of the leaderboard. Good for them, whatever makes them happy.

1453 R, on 12 July 2016 - 05:11 PM, said:

"Hey...Piranha's revised the matchmaking system, like, four hundred and seventy-three different distinct times now and yet we never get to this perfect Utopia of infinite easy nail-biter 12-11 matches everyone keeps b!tch!ng at Piranha over.

Because they keep doing stupid *** ******** ****. Blastman's post is a perfect example of a MM fail. A 5th grader could rearrange those players, using nothing but the stats provided, and do a better job. Another giant fail is PGI's stance on experience = skill, of course their MM doesn't work. I care about match making much more than weapon balance, chassis balance, what mech is OP..... none of that matters nearly as much as having good fights where neither team is set up for failure.

[1453 R]

adamts01, on 12 July 2016 - 08:34 PM, said:

Because they keep doing stupid *** ******** ****. Blastman's post is a perfect example of a MM fail. A 5th grader could rearrange those players, using nothing but the stats provided, and do a better job. Another giant fail is PGI's stance on experience = skill, of course their MM doesn't work. I care about match making much more than weapon balance, chassis balance, what mech is OP..... none of that matters nearly as much as having good fights where neither team is set up for failure.

How do you define 'skill'?

No, seriously. Don't give me that "everybody knows it when they see it" garbage. How do you programmatically, algorithmically define "skill"? How do you program a dumb computer to make nuanced judgments of which players are skilled, which players are lucky, which players are skilled and lucky, and which players are trying to deliberately game the system to earn a 'Skill' rating higher than they naturally warrant (for some ungodly reason)? How do you tell a bunch of lines of code what 'Skill' is, and furthermore!

Since we're indulging in wizardry anyways, how do you guarantee that at any given point in the day or night, any time anyone in MechWarrior Online clicks 'play', there will be twenty-three other people of nigh-identical "skill" levels who have also clicked 'Play' or will click 'Play' inside a fifteen second window, so that each and every player can get the quick, smooth, "perfect" matchmaking all of you guys keep endlessly, fruitlessly, pointlessly pursuing?

Answer: IT'S. A. PIPE. DREAM.

Lemme link you to an excellent piece written by an Overwatch game dev answering this exact same complaint in the Overwatch player base. A game with a hundred times MWO's population and with all the staggering, stupefying sheer manpower and moneypower of ActiBlizzard behind it, and guess what? THEY STILL HAVE A "garbage" MATCHMAKER!

Anyways. Post here: Game Dev Talks Sense.

All right. Go read.

.
..
...
...Caught up? Good. This really lays out all the issues with trying to do Perfect Purely Skill-Based Matchmaking, plain as day. To the point where I'm going to start basically just linking to this article every time some ignorant goober starts kvetching about the matchmaker again.

Short version, for those who're inevitably going to skip the excellent read and just tell me I'm being a Piranha apologist because they can't think: Even Blizzard, with the world's largest non-LoL gaming populations, decades of experience, and effectively infinite resources, cannot do anymuch better in their matchmaking than Piranha can/has/does.

Because when you get human beings involved, predictive models and algorithmic decision-makers can only go so f***ing far.

Edited by 1453 R, 13 July 2016 - 06:52 AM.

[adamts01]

1453 R, on 13 July 2016 - 06:46 AM, said:

How do you define 'skill'?

No, seriously. Don't give me that "everybody knows it when they see it" garbage. How do you programmatically, algorithmically define "skill"? How do you program a dumb computer to make nuanced judgments of which players are skilled, which players are lucky, which players are skilled and lucky, and which players are trying to deliberately game the system to earn a 'Skill' rating higher than they naturally warrant (for some ungodly reason)? How do you tell a bunch of lines of code what 'Skill' is, and furthermore!

TLDR. My answer is that until we have some grand AI, no computer will be able to recognize skill. That's why I stand behind all of my past comments in that PSR should be weighted even more heavily on wins than it already is. I hear a lot of other light pilots argue that "scouting" (looking at the enemy) should be worth more c-bills. I firmly disagree. A computer can't recognize helpful scouting from peeking when you shouldn't and giving away your team's position. That's just one example. So my solution, skip to the end result. Skilled players will win more. They will recognize a weak team and fill in however needed to carry them. Of course not all teams can be carried, but in the end a more skilled player will tip the scale and have more wins.

[MadcatX]

1453 R, on 13 July 2016 - 06:46 AM, said:

How do you define 'skill'?

Answer: IT'S. A. PIPE. DREAM.

Because when you get human beings involved, predictive models and algorithmic decision-makers can only go so f***ing far.

Here's some reading material for you:

http://www.theage.co...0117-hv8uh.html
http://www.wired.com/2010/11/ff_midas/
http://www.marketwir...ing-1951677.htm
https://www.quora.co...rts-bettors-use

To save folks time from reading all the, basically these are all examples of folks using algorithms to determine predictive outcomes betting on sports and the better algorithms go pretty damn far.

Best example I know off-hand are how the top winners from betting sites such as Draftkings are not people who know the sports inside and out, they're programmers.

[1453 R]

MadcatX, on 13 July 2016 - 07:25 AM, said:

Here's some reading material for you:

http://www.theage.co...0117-hv8uh.html
http://www.wired.com/2010/11/ff_midas/
http://www.marketwir...ing-1951677.htm
https://www.quora.co...rts-bettors-use

To save folks time from reading all the, basically these are all examples of folks using algorithms to determine predictive outcomes betting on sports and the better algorithms go pretty damn far.

Best example I know off-hand are how the top winners from betting sites such as Draftkings are not people who know the sports inside and out, they're programmers.

Great.

Now extend those algorithms based on predicting the outcome of matches between a few dozen sports teams, a few dozen times a year, in highly controlled conditions where the players have both the opportunity and the motive to put their absolute, semi-predictable best forward every single match...and extend them to a game with thousands of 'teams' (individual players), at completely unpredictable and uncontrollable times, with no guarantee of any given 'team' performing anywhere near their predicted norm, or even showing up.

Seriously. Read the dev post. If it was so g'damned easy to make this Perfect Beatific Matchmaking Paradise Blastman is pushing for, where every single game you ever play is a 12/11 nailbiter, then it would've already been done. Because Piranha has only gone and fiddled with the matchmaking system something in the realm of fifty-seven thousand times because people like Blastman can't see reality and won't goddamned shut up about it!

[Percimes]

Mister Blastman, on 12 July 2016 - 03:27 PM, said:

The stats do not lie. In them is the truth of a player. The stats can tell us...

If a player hides behind teammates to steal kills and buff their KDR.

If a player is an influencing positive factor in their team's victory and by how much.

If a player can put out damage time and time again.

If a player sacrifices themself for their team over and over but that sacrifice leads to results.

If a player is a chronically bad player.

The stats tell us many things and one thing they do not lie about is if a player is truly bad.

The stats are too young to speak any truth for most people. At this point, what you get from them is what you want to find. At the end of the summer you may have a point, but right now too many people don't have enough matches played to compare people.

I guess if you only (or mostly) play during prime time it's hard to grab the randomness of the off-hours when the valves are opened and you get all kind of potatoes, from the delicious fries who can shoot to the mashed ones who pool themselves in the various holes and depressions of the maps.

As for the part I bolded: doing the right thing for your team doesn't mean much if no one use the opportunity, which is what happen in the vast majority of the time I've done it, which is not a rare thing. QP in the off-hours is more like 3 vs. 21 than 12 vs. 12.

[Brain Cancer]

If the current MM has to deal with everyone from dunce-dipsticks to secret Terminators being in the same pool, it can't match anything. Computers do what they're told. If they're told to sort by tiers and a tier is nothing but an exp bar, you just get a random mix of actually skilled players with bad players who just happened to play lots and get carried enough to level up. It's a handful of candied M&Ms for the MM where it can't tell which have chocolate and which have poop in the center because all it's told to do is give each team 12 M&Ms of roughly the same size and color (weight and tier). Chow time.

[Littlerift]

adamts01, on 12 July 2016 - 08:34 PM, said:

A 50% W/L and 1:1 K/D would be awesome. I'd take 10 great losing fights over 10 winning stomps any day or the week.

Hmm, okay. I don't feel the same but fine. This question comes up a lot in WoT and the general consensus is that skill-based matchmaking will kill the game: skilled players will be put off by the fact that their stats will plummet with skill-based MM, and poor/new players will never learn because they'll never fight anybody good.

Not to mention the fact that in a perfect skill-based system you inevitably end up (eventually) in a situation where it's impossible to tell using statistics which players are good and which are bad, and so eventually everybody ends up together anyway.

Edited by Littlerift, 13 July 2016 - 01:02 PM.

[EgoSlayer]

MadcatX, on 13 July 2016 - 07:25 AM, said:

Here's some reading material for you:

http://www.theage.co...0117-hv8uh.html
http://www.wired.com/2010/11/ff_midas/
http://www.marketwir...ing-1951677.htm
https://www.quora.co...rts-bettors-use

To save folks time from reading all the, basically these are all examples of folks using algorithms to determine predictive outcomes betting on sports and the better algorithms go pretty damn far.

Best example I know off-hand are how the top winners from betting sites such as Draftkings are not people who know the sports inside and out, they're programmers.

Key difference there is that you are plotting the behavior of professional athletes who are trying to win every game they play, with teammates they practice with for hours everyday and are getting paid to do this and often have incentive based pay based on their performances.

That is so much like MWO where you have 24 random people thrown together who likely are not all playing meta builds and or are playing for lols/fun only. Yeah, 100% relate-able.

Maybe it would work for comp teams playing in tournaments and competitions, but for general pug queues? No way. The other thing is the big data and computations that are going into these sims, some can be crunching numbers for days to get results, nobody is going to wait that long for matches. It has to be done in seconds which is why the match makers are as simple as possible.

Edited by EgoSlayer, 13 July 2016 - 05:58 PM.

[Brain Cancer]

adamts01, on 12 July 2016 - 08:34 PM, said:

A 50% W/L and 1:1 K/D would be awesome. I'd take 10 great losing fights over 10 winning stomps any day or the week.

But I got that even in the current year matchmaker.



The current "matchmaker" is really just hiding that they gave up entirely on actually dividing people by skill. The game just wants you to play lots so you get your big number "1" and will never stop adding increasingly terrible players to any given tier until eventually, we all T1's, yay best players!

[habu86]

I haven't read the last 5 pages of this thread, so apologies if someone's pointed this out already, but while the idea certainly has merit and has served to actually get people to supply real data to a debate that really, really needs it, but, from a methodology standpoint, there are couple of problems with the way we're analyzing this.

1.) Selection bias. By focusing only on those matches that went horribly sideways, we're essentially only looking at those instances where MM failed hard. Now, there is definite value in doing that, as it's those exact instances that are most frustrating and whose root causes need to be addressed. But evaluating the quality of MM based on just those instances where it fails may paint a gloomier picture than is really the case. The danger to making an overly gloomy assessment lies in assigning a higher priority for the allocation of limited resources than is really called for. Which is not to say that it's not frustrating to be on the receiving end of a string of losses, and, on a night where you somehow go 1-10, it's easy to remember all those losses as horrible wipes, regardless of whether they were close or not (and some regions are worse off than others - looking at you Oceanic...), which feeds directly to disincentivize players from continuing with the game.

2.) Evaluation methodology. By looking only at averages, we're falling into the same trap as PGI's MM when evaluating teams. Player skill distribution matters. For example, 2 comp-grade players may try, but ultimately fail to carry 10 terribads against 12 players of average and just-above average skill, regardless of how close averaging out their "skill metric" tells you the teams should be. By averaging out PSR (regardless of how they calculate it) and seeking to minimize the difference between averages, MM is sweeping some genuinely important considerations under the rug. My personal opinion is that attempting to minimize a distance metric (such as the Kolmogorov-Smirnov statistic) between the two distributions would likely lead to better MM outcomes in terms of skill assignment, but carries the risk of increases in required computational time.

Lastly, I won't go into all the other stuff that can feed into match outcomes such as individual mech loadouts, player familiarity with individual classes, and so on, as others have already done a very good job of discussing these issues already. Either way, good job to the OP for getting this conversation started and to everyone who's taken the time to contribute data. It's a sorely needed initiative.

Edited by habu86, 13 July 2016 - 07:32 PM.

[adamts01]

Littlerift, on 13 July 2016 - 01:02 PM, said:

Hmm, okay. I don't feel the same but fine. This question comes up a lot in WoT and the general consensus is that skill-based matchmaking will kill the game: skilled players will be put off by the fact that their stats will plummet with skill-based MM, and poor/new players will never learn because they'll never fight anybody good.

Not to mention the fact that in a perfect skill-based system you inevitably end up (eventually) in a situation where it's impossible to tell using statistics which players are good and which are bad, and so eventually everybody ends up together anyway.

I find stats interesting but I couldn't really care less about them. I play this for fun, and for me that means having a good challenging fight. Putting stats before fun and needing to see your name on top of the leader-board seems like an indicator of some confidence issues to be. As for those punks in WoT, they're just like the Seal clubbers in CW. They go out, see who can get the most kills and then think they're the ****, even though they went up against noobs as a 12-man. They don't want a fair fight, or really any challenge where they have a good chance at losing. They have to win, it's pathetic.

As for new players not learning, true they need a challenge, but they also need to be able to try out the things they're learning. If I sparred a pro fighter I'd get my *** kicked before I knew what happened. I really believe it helps you improve competing against an equal or slightly better opponent, so for each side to benefit, competitors should be equal.

[Brain Cancer]

http://www.sarna.net/wiki/Komodo

11E (10 ML + a TAG), 3053 build date.

You'd have one legal variant with the KIM-2A and then you'd have to go into -P like the Panther and Enforcer.

[Drenath]

Another way to make closer games might be emergent handicaps based on killscore. Variable crit chance, auto-UAV's, reduced heat damage/shutdown time, etc. Subtle, increasing assistance to a team more than 2 kills behind. Definitely need to be wary of it being abused though.