1911 replies to this topic

[SnagaDance]

Karl Berg, on 19 July 2014 - 02:16 PM, said:

Yup, this has been slightly modified for a few reasons. The old algorithm is pretty much as I described earlier in this thread.

There was never any consideration of Elo in lance assignment, it has always been driven off of group size. This is due to the requirement that we split groups as infrequently as possible.

Previously, this assignment of players to lances was done on the dedicated server. Last patch, I deleted this dedicated server code, and re-implemented assignment to happen on the matchmaker itself. This resolves a potential bug, where the dedicated server could clobber lance assignment done in private lobbies.

The new algorithm on the matchmaker is simply this:

- Sort requests by group size
- Select a random seed lance
- For each request
   - Increment seed lance
   - For each user in request
	  - While seed lance is full
		 - Increment seed lance
	  - End
	  - Place user into seed lance
   - End
- End

This correctly handles all manners of input group sizes with very little code. Randomizing the seed lance means the largest group is no longer assigned to Alpha lance 100% of the time. Hopefully now you will see a much larger variety of initial spawn points in game if you tend to play with a specific group size. Also, no more bias against Charlie lance.

Very interesting. I now assume that with the old system my Alpha lance pick was probably based on how long I've actually been playing or how long my Pilot name has been registered. In essence I simply continued playing while others dropped out. It surely isn't for my skills that I was in Alpha that much while at the same time Charlie's players were very evidently newcomers.

[Kovach]

TheCaptainJZ, on 23 July 2014 - 08:31 PM, said:

From a logic point of view, you don't want the matchmaker to make a decision based on a decision (cbill rewards) based on the variable you really want the matchmaker to look at. You'd go straight to those variables so the two would end up be complementary but one should not result from the other.

Logic only does help if you ask the right questions.
Do I want the matchmaker overly complex - no.
Does every pilot have collection of the current match actions - yes.
Does these actions (for current match) are processed anyway - yes.
Can the values of pilot actions be adjusted to reflect the player skill - very possibly.
Since the so called "variables" are always collected and even processed in the form of money, it will not cause any overhead to do a simple calculation only on that variable. The big problem is the balance between the monetary gain by doing different roles, but that is entirely possible.

My question is, do you prefer a simple (possibly fast and deterministic) algorithm for evaluating player ability more based on OTHER players
or
you prefer a system that is a bit more complex (requiring balance of rewards, but also deterministic) that matches pilots based on their OWN performance?

My logic is that since the variables are there, it's best to use them, not just ignore them.
Does not every pilot deserves to be "teamed with" and "against" pilots performing similarly? My logic sais yes.

[Jman5]

Hey Karl, what ever happened to making Zoom client side? I'm getting latency in my zoom feature again and it doesn't work in games that lag out. I thought this was changed several patches back.

[Void Angel]

Kovach, on 24 July 2014 - 10:54 AM, said:

Logic only does help if you ask the right questions.
Do I want the matchmaker overly complex - no.
Does every pilot have collection of the current match actions - yes.
Does these actions (for current match) are processed anyway - yes.
Can the values of pilot actions be adjusted to reflect the player skill - very possibly.
Since the so called "variables" are always collected and even processed in the form of money, it will not cause any overhead to do a simple calculation only on that variable. The big problem is the balance between the monetary gain by doing different roles, but that is entirely possible.

My question is, do you prefer a simple (possibly fast and deterministic) algorithm for evaluating player ability more based on OTHER players
or
you prefer a system that is a bit more complex (requiring balance of rewards, but also deterministic) that matches pilots based on their OWN performance?

My logic is that since the variables are there, it's best to use them, not just ignore them.
Does not every pilot deserves to be "teamed with" and "against" pilots performing similarly? My logic sais yes.

Logic also requires that you start from the proper assumptions. To quote an unconsciously ironic quip from Asimov, "You can prove anything you want by coldly logical reason---if you pick the proper postulates." You're using a faulty assumption to lay out your argument: you've started by pointing out that using c-bill awards is a possible method (actually experience is better, but I digress) that won't generate excessive overhead for the servers. This is true, but you move on to assume a false dichotomy: that the choice is simply between using team stats to determine ranking, or more personalized scoring.

This is incorrect: the choice isn't that clear-cut or simplistic. What we have to choose from is a simple system which will, for the vast majority of players, correlate ranking to skill in the long run; or else a more complex (and therefore uncertain system) based on game rewards which are currently not balanced - and which will result (even if the c-bill rewards were to be balanced) in players with the same skill level having different rankings in the long run if their equally successful playing styles approach the game in ways different from the design of the reward system.

Elo is used for a reason; it is not an arbitrary decision on the part of PGI, and there is no alternative system which does not have the kind of pitfalls I just outlined.

Edited by Void Angel, 24 July 2014 - 02:51 PM.

[p4r4g0n]

Kovach, on 24 July 2014 - 10:54 AM, said:

-snip-
Can the values of pilot actions be adjusted to reflect the player skill - very possibly.
-snip-

This is an extremely dubious assumption. In order for it to be valid, it would have to be presumed that despite PGI previously indicating that they are not entirely happy with the current rewards system and acknowledging that it does NOT reflect player contributions in a match as completely as they would prefer, they either do not care or do not have a clue as to how to improve it. (Note: In case you were not aware of it, this issue has actually been discussed extensively in the past).

Kovach, on 24 July 2014 - 10:54 AM, said:

-snip-
My question is, do you prefer a simple (possibly fast and deterministic) algorithm for evaluating player ability more based on OTHER players
or
you prefer a system that is a bit more complex (requiring balance of rewards, but also deterministic) that matches pilots based on their OWN performance?

My logic is that since the variables are there, it's best to use them, not just ignore them.
Does not every pilot deserves to be "teamed with" and "against" pilots performing similarly? My logic sais yes.

The mis-matches that you see under the current system is NOT due a fundamental flaw in the way in which players are rated. It is due to the fact that at a given instant in-time there are not enough players queuing that have the exact same player rating to fill 24 slots in the solo queue. This issue is exacerbated for groups due to the averaging of the ratings of all the players in the group to arrive at a group rating for match-making purposes.

In order to achieve the perfect match-ups that most of us would probably like see, it must be acknowledged that:-

• much longer wait times will be needed per match and grow progressively longer as a player's rating increases (or decreases) away from the average
  
• outlying high / low rated players as well as those playing during non-peak times will definitely be unsuccessful in obtaining a match in any reasonable amount of time
  
• pre-made groups have to be removed entirely

TL;DR

Your proposal with regard to player ratings will not address the main issue of mis-matched players in a team as your solution does not eliminate the fundamental match-making constraints that create these mis-matches.

Edited by p4r4g0n, 24 July 2014 - 08:18 PM.

[p4r4g0n]

Hi Karl,

A quick question regarding the new MM if I may.

I have been told by someone I regularly drop with that he encountered another mutual acquaintance who was in a group group (and confirmed this) while dropping in the solo queue. This happened 2-3 days ago after Russ posted that the solo queue was 100% solo only.

Have you guys been altering the MM parameters to gather additional data and is this going to be on-going for a while or has there been a decision to revert to the concept of 1 group on each team in the solo queue?

Edited by p4r4g0n, 24 July 2014 - 05:33 PM.

[Siriothrax]

Hey Karl, any update on the mg desync issue? :3

[Wintersdark]

Cimarb, on 23 July 2014 - 07:44 AM, said:

That is actually common, and what I refer to as "average horror", where the average of High+Low Elo players is assumed to be the same as the average of a bunch of Medium Elo players. It is the whole reason I dislike this whole system. If you average numbers enough, EVERYTHING is average, and that makes the individual experience horrible.

This is why I've asked many times what the average Elo delta is within a team.

They've always been quite up front with team vs. team deltas... But the spread within a team is a very interesting question.

You're absolutely right though, that 2 strong+2 weak players != 4 average players, or a good experience at all, really. However, I can't comment on how that actually is without knowing what the delta really is. After all, everyone likes to think that they guy on their team who accomplished nothing or did something really stupid is just bad, but there are countless reasons why a legitimately good player may totally mess up, so you can't really know.

[Wintersdark]

Void Angel, on 24 July 2014 - 02:48 PM, said:

Elo is used for a reason; it is not an arbitrary decision on the part of PGI, and there is no alternative system which does not have the kind of pitfalls I just outlined.

This is what I always go back to.

We are all aware of the downsides to Elo. It is not a perfect system by any means.

However, simply stating "PGI just needs to build an admittedly more complex system that uses player stats to determine their skill level" is neither fair nor realistic.

Ranking player skill is an immensely hard task, in any game. It sounds easy, or at least manageable, but it's simply neither. Look at how hard it is to get rewards right, how can you rank a player's contributions to a match in a manner that cannot easily be gamed for profit?

Elo, for all it's flaws, is difficult to game and certainly not profitable to game. Elo actually accounts for everything, and is self-balancing over time.

A stat or reward based system needs to be manually tuned, and cannot account for even a fraction of the things which contribute to victory or defeat. How do you rank kills vs. assists? If you rank kills higher, I can just cripple mechs and leave them - a strong contribution to victory - while the bottomfeeder who just grabs easy kills and hides otherwise will end up ranking higher than me... and thus facing stiff competition while my foes just get weaker and weaker. It's not technically impossible, but it's extremely difficult. So much so, that I can't think of a single even moderately complex game that's ever managed to get a non-Elo based system to actually work reasonably well.

There's a very good reason PGI went with Elo.

[TheCaptainJZ]

I'm convinced that Elo is the way to go for matchmaking in the long run, but as many players have complained, people can get long losing streaks (and by correlation, long winning streaks, but no one complains about those). Can anything be done about that? If the MM predicts you are going to lose and you do, it doesn't adjust your Elo (at all or very little). What if it did lower it some? Would that break the system? I don't know the cause of streaks, but maybe it's partially caused by player being projected to lose and they do for several matches in a row. If their score lowered each time, they would get paired up with weaker opponents faster and theoretically win a round or two quicker, but does this make the system easier to game?

[Kovach]

Wintersdark, on 24 July 2014 - 11:37 PM, said:

...
Ranking player skill is an immensely hard task, in any game. It sounds easy, or at least manageable, but it's simply neither. Look at how hard it is to get rewards right, how can you rank a player's contributions to a match in a manner that cannot easily be gamed for profit?
...

It is unfair solo drops (matched by semi-random people) to be judged same way as you were in a constructed team (which deserves that judgment).
If the system is not simple by nature (12 players, not 2) you might use the same methods to analyze it, but the results can be very unfair.

If it's impossible or not manageable to rank solo drop players, then it won't be done - end of story.
If it's possible, it will not make match making faster, but maybe more fair for solo drops.

[Shlkt]

Quote

You're absolutely right though, that 2 strong+2 weak players != 4 average players, or a good experience at all, really.

But the Elo system still works in this case; eventually the Elo ratings of strong players will drop in order to compensate for the matchmaker behavior. In other words, if you can't carry a team by yourself and yet the matchmaker assumes that you can, then eventually Elo's failed predictions will ensure that your rating drops to compensate. The matchmaker will group you with fewer "bad" players as a result.

The following chart shows the relationship between two Elo ratings for simulated pilots. One rating was determined using a "restricted" matchmaker which tries to form games such that all participants are within a narrow Elo band; essentially everyone is playing against pilots (allegedly) similar to themselves in skill level. The second rating was determined using a random matchmaker (i.e. high Elo deltas) which tries to keep teams even but does not restrict participation based on skill level.



Notice that the so-called "random matchmaker", which is probably more appropriately named high-delta matchmaking, results in narrower range of Elo ratings in general; also, the relationship is definitely non-linear; flattened regions of the curve at either end indicate that exceptional pilots (either exceptionally good or exceptionally bad) are affected the most by the transition. Essentially the great/awful pilots have less of an impact on the outcome of the match when teams are random (this is because there's a strong possibility that the other team has an equally bad/great pilot).

Whether "high-delta matchmaking" is good or bad for the overall player experience is debatable, because the alternative is either incredibly long queue times or unbalanced weight restrictions like we had before. Those downsides are, unfortunately, inevitable without an extremely large number of concurrent players in the queues.

Edited by Shlkt, 25 July 2014 - 08:00 AM.

[Cimarb]

Wintersdark, on 24 July 2014 - 11:37 PM, said:

This is why I've asked many times what the average Elo delta is within a team.

They've always been quite up front with team vs. team deltas... But the spread within a team is a very interesting question.

You're absolutely right though, that 2 strong+2 weak players != 4 average players, or a good experience at all, really. However, I can't comment on how that actually is without knowing what the delta really is. After all, everyone likes to think that they guy on their team who accomplished nothing or did something really stupid is just bad, but there are countless reasons why a legitimately good player may totally mess up, so you can't really know.

I have had some truly terrible matches were I just plain screwed up, and it had nothing to do with my team, and also matches where my team left me high and dry and my score (or lack therof) showed that, so I completely agree.

I think team Elo spread means little without knowing the inner delta, like you said. You cannot just look at averages of averages and think they are a good representation of individual match quality.

Wintersdark, on 24 July 2014 - 11:51 PM, said:

A stat or reward based system needs to be manually tuned, and cannot account for even a fraction of the things which contribute to victory or defeat. How do you rank kills vs. assists? If you rank kills higher, I can just cripple mechs and leave them - a strong contribution to victory - while the bottomfeeder who just grabs easy kills and hides otherwise will end up ranking higher than me... and thus facing stiff competition while my foes just get weaker and weaker. It's not technically impossible, but it's extremely difficult. So much so, that I can't think of a single even moderately complex game that's ever managed to get a non-Elo based system to actually work reasonably well.

The XP/cbill reward system already "ranks" us on an individual basis. While it needs improvement, such as much more granular spotting and capping rewards, it does a pretty good job of showing your contribution to the match. I think a "leadership" award based upon the number of mechs remaining on your team would be great too, but regardless...

Elo is supposed to take all of those items into account, since they all contribute to a win or loss, but it does not do that very well. You can average ANYTHING and it will eventually be averaged. Along the same lines, do you want to only get an XP/cbill reward if your Elo goes up? Of course not, because that would be a horrible system! You want to be rewarded for your individual achievements, whether your team succeeded or not, but the bonus for your team winning is icing on that cake too. So it should be for rating players, IMO.

[Wintersdark]

Cimarb, on 25 July 2014 - 08:09 AM, said:

The XP/cbill reward system already "ranks" us on an individual basis. While it needs improvement, such as much more granular spotting and capping rewards, it does a pretty good job of showing your contribution to the match. I think a "leadership" award based upon the number of mechs remaining on your team would be great too, but regardless...

The difference is, we are rewarded based on arbitrary criteria. That criteria can be (and is, incidentally) terrible for actually determining player skill and as such is worthless for matchmaking, and that's ok, because someone skewed or oddball reward amounts really are not a major concern: Devs can simply tune the economy based on average earnings per match, and go with that, even if they are "unbalanced".

On the other hand, using those rewards to determine player skill is fraught with peril, because those rewards do not reflect how much you contributed to winning the match. Period. They're rewarding actions that are generally positive, but they are also highly exploitable (either way) and.

Quote

Elo is supposed to take all of those items into account, since they all contribute to a win or loss, but it does not do that very well. You can average ANYTHING and it will eventually be averaged. Along the same lines, do you want to only get an XP/cbill reward if your Elo goes up? Of course not, because that would be a horrible system! You want to be rewarded for your individual achievements, whether your team succeeded or not, but the bonus for your team winning is icing on that cake too. So it should be for rating players, IMO.

See, you keep looking at Elo in these conversations as a reward (if it increases), or a punishment (if it drops). This is flat out incorrect. Elo is not a rating of your individual skill, it's a matchmaking parameter that strives to build teams of players who overall contribute a comparable amount to team victory.

Obviously then, I wouldn't want to be rewarded based on Elo change, because that would result in my not getting match rewards simply because the opposing team was better than me. Conversely, I don't want my Elo rating to go up, matching me against more skilled players, simply because I got extra assists but didn't contribute more to team victory, because that would result in poor play being "rewarded" by facing against even stronger opponents.

High match results are not indicative of strong play. You don't win in MWO by being a good shot, or by doing lots of damage. You win first and foremost by cooperation and teamwork. This is always true; always. I see losing teams with the most damage done overall all the time. Elo is incorporating this as best it can to bring matches where the teams ideally have a 50:50 chance of victory. That's all it cares about. And, as Karl has said, it's actually pretty good at this overall.

So, here's the thing.

We'd all love to see a system that works better than Elo. I don't think anyone in this thread is particularly amazed by how well Elo works.

But, in the end, it's not good enough to say "I want to see matchmaking based of stats". We have stats, and nobody knows how to interpret them to actually measure player skill. You're just wishlisting there, and it's totally useless. My challenge to you: Design a system, with specific numbers, that actually ranks players. Start simple, give me an example. How would YOU rank damage vs. assists vs. kills vs. spotting vs. narc/tag? Can you do this in such a way as to not rank a very skill player low (I'm very precise, and kill or disable mechs with minimal damage done) or rank a low skilled player high (I spray and pray, do a fairly large amount of damage to many mechs but don't actually do any damage that's useful to my team)?

Because the issue is quite simply that what you feel that PGI should be able to do is extraordinarily hard and maybe even practically impossible - it certainly hasn't been done successfully yet.

This is comparable to saying "Hey, chemotherapy and radiation treatment are a terrible way to deal with cancer, sometimes even causing more harm than good. Doctors should just come up with a cure instead."

[TheCaptainJZ]

Think of the formula used in tournaments:

Quote

• Score Formula:
  Kills × 20 + Kill Assist × 10 + (Damage Done – Team Damage) ÷ 15 + Wins × 20 + Loss × 5
  
• 1^st Tie-Breaker: Number of Matches Played (Less the better)
  
  • EG: My best 10 matches out of 20 is better than your best 10 matches out of 50
• 2^nd Tie-Breaker: Team Damage (Less is better of course)
  
  • EG: My 10 best matches with 100 team damage total is better than your Best 10 matches with 150 team damage total.

This led to even more players gravitating towards meta mechs and meta builds. Does this reflect skill? I think it reflects endurance more (it picks the best out of 10). You could have had 10 good games and 100 bad games and if a matchmaker used something like this to match you, it would probably not call you a top player. No one is saying Elo is great, but no one has a better alternative with real numbers.

[Karl Berg]

p4r4g0n, on 24 July 2014 - 05:33 PM, said:

Hi Karl,

A quick question regarding the new MM if I may.

I have been told by someone I regularly drop with that he encountered another mutual acquaintance who was in a group group (and confirmed this) while dropping in the solo queue. This happened 2-3 days ago after Russ posted that the solo queue was 100% solo only.

Have you guys been altering the MM parameters to gather additional data and is this going to be on-going for a while or has there been a decision to revert to the concept of 1 group on each team in the solo queue?

Nope, we haven't touched any matching rules since shortly after the last patch. In general the solo games the matchmaker is producing look pretty healthy overall. For the solo queue, statistical ranges are down, average Elo per team is generally within 50 units despite a good range of average Elo games (ranging anywhere from ~600 Elo at the low end to ~1700 at the high end).

Siriothrax, on 24 July 2014 - 08:40 PM, said:

Hey Karl, any update on the mg desync issue? :3

Thanks for the reminder!

Yes, we do seem to have good news. While the issue is not completely resolved, there are some very focused fixes that should make it in for next patch that seem to provide a very noticeable improvement with respect to this issue.

[Siriothrax]

Karl Berg, on 25 July 2014 - 02:42 PM, said:

Nope, we haven't touched any matching rules since shortly after the last patch. In general the solo games the matchmaker is producing look pretty healthy overall. For the solo queue, statistical ranges are down, average Elo per team is generally within 50 units despite a good range of average Elo games (ranging anywhere from ~600 Elo at the low end to ~1700 at the high end).

What does the Elo variance within the teams look like for those games? Just by feel, it seems kinda high.

[Void Angel]

Karl, could I get you to confirm something I'm having trouble sourcing from the forums - Do critical hits which do not have a valid target (e.g. empty side torsos, or equipment previously destroyed) still deal the 15% extra damage?

Siriothrax, on 25 July 2014 - 08:46 PM, said:

What does the Elo variance within the teams look like for those games? Just by feel, it seems kinda high.

You mean the standard deviation from the team average?

[Modo44]

Maybe he meant variance. I asked about this a couple of months ago, and I do not recall a straight answer.

[Void Angel]

Standard Deviation is a form of variance. I'm wondering what number he's actually asking for. Variance is kind of broad unless you specify type.

Also, standard deviation is related to the kind of numbers being actually given to us.