277 replies to this topic

[Kyle Polulak]

With the recent news from the following Command Chair Announcement:
http://mwomercs.com/...ost__p__2265319

Spoiler

Omid Kiarostami, on 19 April 2013 - 01:08 PM, said:

We’ve done our tuning pass on the Matchmaker and have some good results to report. It’s hard to talk about this without going into details, so I’m going to get a little technical and share some of the data we’re looking at – just scroll down to the summary if you want the cliff notes.


ELO ADJUSTMENT

The Elo adjustment went in beautifully. To show you what we’re looking at, this is our distribution of players across Elo values:


For the curious, the darker portions of the graph indicate players who have played more than 50 matches. Visually, it indicates how the curve distributes itself over time.

The graph below is what our distribution was like before:


You can see how this curve favored people who had played more matches, distributing players across a much broader range. When we moved from the red curve to the blue one, we started measuring extremely positive results on our servers.

In my last post, I shared a statistic: That 74% of our matches were within “tolerable” weight balance. When we started matching with the blue curve, that number spiked to 88%. Beyond that, less than 2% of our matches now fall into the “horrendously bad” category we described earlier – that’s down from the over-7-percent number it was before. The difference in Elo for most matches produced is also is less than half of what it used to be.

We tried tuning the weight class tolerance down to further improve this, but got mixed results. The trade-off was between an improvement to average match quality (which has already changed with the Elo adjustment) and causing more of our outlying groups to fail to find matches entirely (specifically the 8-man queues). We’re not satisfied with the state of this, and have left the tolerance wide enough for 8-mans to continue to match while we work out a more complete solution. Note that our considerations include using tonnage as a direct factor to matchmaking, among other things.


IN SUMMARY

The result of all this is that we were able to cut down the number of unbalanced matches, but not eliminate them entirely. To summarize:

• There’s better skill matching of teams in game.
  
• Uneven weight balance matches occur less frequently.
  
• Wait times are lower (thanks to a denser curve).

This is something that we are monitoring heavily and will continue to address over time. As always, we appreciate the discussion and comments. You can more talk about this follow-up here.

This thread shall serve as the official response channel.
Please keep your feedback, responses, comments and concerns limited to this thread so that we may both consolidate your responses and ensure no valuable opinions are lost.

Please also remember to keep your responses in line with the Code of Conduct and direct your comments to us (PGI/IGP) rather than arguing with other respondents. Let's keep it constructive!

[Lazydrones541]

So far I must say I am quite pleased with the way the Elo is working now as it no longer seems like a fat kid on a see-saw.
Havent noticed but maybe 1 game in which the teams were horribly unbalanced (we still won - the lighter side.)
Only a few games of noticeably uneven KD matches. Other than that they seem to be reasonably even.
(Most seemed to be within 2-3 left alive on either team; aside from of course cap games)
So far, a good job indeed.

Edited by Lazydrones541, 19 April 2013 - 01:29 PM.

[Capt Cole 117]

Reserved for witty comment.

[5th Fedcom Rat]

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.

[WardenWolf]

My biggest concern left with matchmaking is the experience of new players. The light blue on the Elo chart is people with less than 50 games - aka new players. They are right there in the middle of the distribution, which means you are dropping new players in with 'average' veteran players... which means about half of those new players (those destined for lower Elo eventually) will struggle and may bail on the game if they have bad experiences early-on (matched against better players + stuck in trial mechs).

I would still suggest having the newer players come in at a lower Elo area, rather than right in the middle. I'd be interested in seeing how that might help the new player experience, and what it might do to the distribution graph as well.

[BumRuckus]

I think that weight balancing is less of an issue than people percieve it to be. I notice among some players I am matched with, when we loose badly they immediately blame it on a "sync-drop" or being "out-tonned." I think this is a fallacy and it's harder for people to step back and say, "we just made bad decisions," or "we were out-played."

To be quite honest, I like the mismatched weight balance in games, it allows for a much greater variety in game play. When the games are too equally matched all the time, they get boring. When I'm in my Raven 2X, I want to play against heavies and assaults sometimes, not always other light-squads.

To me, the most important thing to match is skill. A good ELO matching is going to give people the best games no matter what. A highly skilled heavy team vs a highly skilled lighter team is going to be a good match. Period. This is especially important in 8-mans. A team should be allowed to make their OP ideal composition, be it 8 raven 3Ls or 8 Atlas DDC's, and still be matched against equally skilled players. If everyone wants to brawl in the middle of the map against exactly the same weight team, they might as well flip coins all day on their desktop.

Edited by BumRuckus, 19 April 2013 - 02:17 PM.

[armyof1]

WardenWolf, on 19 April 2013 - 02:02 PM, said:

My biggest concern left with matchmaking is the experience of new players. The light blue on the Elo chart is people with less than 50 games - aka new players. They are right there in the middle of the distribution, which means you are dropping new players in with 'average' veteran players... which means about half of those new players (those destined for lower Elo eventually) will struggle and may bail on the game if they have bad experiences early-on (matched against better players + stuck in trial mechs).

I would still suggest having the newer players come in at a lower Elo area, rather than right in the middle. I'd be interested in seeing how that might help the new player experience, and what it might do to the distribution graph as well.

This is exactly what I've been thinking, new players should not be dropped right down into mid-tier players as most of them would be basically punching bags. It's not fun for them and it's not fun for the experienced players to have them in their teams either. Looking at the blue graph I'd say drop them in at around 1050 as they won't have trouble finding matches there and it's opponents that won't blow up their cockpits on a regular basis.

Edited by armyof1, 19 April 2013 - 02:45 PM.

[Mystere]

5th Fedcom Rat, on 19 April 2013 - 01:43 PM, said:

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.

Given that I am really leery of numbers just being thrown around, I'd like to request that you provide more details on the highlighted "facts".

Edited by Mystere, 19 April 2013 - 02:24 PM.

[Chemie]

The reason you are getting better weight balance is not the ELO change. It is because everyone and their dog is dropping in highlanders right now so there is less diversity of mech class in the queue. There is no reason that changing ELO would cause better weight balance. You are implying that the same skilled players drop in the same chassis which is stupid.

Also, forcing weight balance in 8-man is crazy because it is so empty. You need different tolerance on 4-man and 8-man queues.

Edited by Chemie, 19 April 2013 - 02:31 PM.

[Zylo]

PGI needs to look at why matches are "balanced". As others have said before it's probably a result of so many assault mechs in use. PGI should look at the weight class distribution in "balanced" matches. Does each team have nearly all assault mechs in a "balanced" match while a group taking lights + mediums ends up in a very imbalanced match with no lights or mediums on the enemy team?

[Glucose]

I just want to say - great follow up post on the ELO matchmaking. I loved the graphs and the visualization. It's very reassuring that you're taking real measurements and presenting them in this sort of fashion.

I personally, noticed a huge improvement in my teammates skill in the past few days playing. I could tell the matchmaking was working at a higher level.

Thanks for the hard work.

[Devil Fox]

Zylo, on 19 April 2013 - 03:53 PM, said:

PGI needs to look at why matches are "balanced". As others have said before it's probably a result of so many assault mechs in use. PGI should look at the weight class distribution in "balanced" matches. Does each team have nearly all assault mechs in a "balanced" match while a group taking lights + mediums ends up in a very imbalanced match with no lights or mediums on the enemy team?

That's how it happens... everyone is assault heavy, and their crawling all over the shop in assault mode. So Light's are migrating to conquest... had 7 Cicada and below mechs and one Jaegermech vs 3 assault, 3 Heavy, and 1 each of the fast medium/light... seriously out-tonned but we still killed 5/8 of their team in a open brawl.

The one problem is that I can still be waiting for a little while for a game, now you say the teams are roughly skill balanced but the elo seems to quite happily throw itself open to putting in trial mechs into games, and majority of those continue to disconnect.

[Jetfire]

5th Fedcom Rat, on 19 April 2013 - 01:43 PM, said:

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.

Only time will tell, but it certainly didn't hurt the numbers.

[Kageru Ikazuchi]

The comparison between the two graphs makes perfect sense to me ... and this is why:

• I have played about 1000 matches since you started seeding Elo scores (in mid February)
  
• since about mid-March, my matches had gotten progressively harder (and my stats kept getting worse)
  
• since you re-computed Elo (I'm guessing my scores all went down, except Assault - I almost never play assault) the matches have gotten much easier, and my stats appear to be trending up (I'll know for sure Tuesday, I track them weekly)

So I think I'm "rubber banding" a bit with my Elo scores, but they should settle down in a week or so.

Thanks for the clarity and communications.

[Omid Kiarostami]

Mystere, on 19 April 2013 - 02:23 PM, said:

5th Fedcom Rat, on 19 April 2013 - 01:43 PM, said:

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.

Given that I am really leery of numbers just being thrown around, I'd like to request that you provide more details on the highlighted "facts".

I like the way you think, Mystere.

I can step in and share something on that: We measured a 10% increase in the total number of players matching with Assault-class Mechs, relative to the other weight classes. I don't believe an increase of this nature would cause the weight balance of matches to improve like this, though - if anything, if it were large enough to be significant, it might cause things to get worse by creating more outlying matches (with the other weight classes) that would be harder to balance. It's tough to say.

The point is kind of academic, though; the reason weight balance has improved is entirely different. It's behaving better because the match maker no longer has to stretch across the entire breadth of the Elo curve in order to find matches.

To break it down, what's significant about the new distribution is that there's now something around a 500 point Elo delta containing the majority of players in our system. On the old curve, you can observe a clear 1300-point gulf splitting the player base in two.

This is important if you know that the constraints on Elo and Weight are broadening together with time. So previously, by the time matchmaker had reached across the curve to pick up some more players it had already lowered it's standards on weight significantly. Now that it doesn't need to do this it's finding better matches faster.

So in short: Better data is allowing the match maker to provide better results.

Edit: I posted this before reading Thontor's reply. He summed it up pretty well for me.

Edited by Omid Kiarostami, 19 April 2013 - 07:53 PM.

[hammerreborn]

Omid Kiarostami, on 19 April 2013 - 07:38 PM, said:

I like the way you think, Mystere.

I can step in and share something on that: We measured a 10% increase in the total number of players matching with Assault-class Mechs, relative to the other weight classes. I don't believe an increase of this nature would cause the weight balance of matches to improve like this, though - if anything, if it were large enough to be significant, it might cause things to get worse by creating more outlying matches (with the other weight classes) that would be harder to balance. It's tough to say.

The point is kind of academic, though; the reason weight balance has improved is entirely different. It's behaving better because the match maker no longer has to stretch across the entire breadth of the Elo curve in order to find matches.

To break it down, what's significant about the new distribution is that there's now something around a 500 point Elo delta containing the majority of players in our system. On the old curve, you can observe a clear 1300-point gulf splitting the player base in two.

This is important if you know that the constraints on Elo and Weight are broadening together with time. So previously, by the time matchmaker had reached across the curve to pick up some more players it had already lowered it's standards on weight significantly. Now that it doesn't need to do this it's finding better matches faster.

So in short: Better data is allowing the match maker to provide better results.

Edit: I posted this before reading Thontor's reply. He summed it up pretty well for me.

Hey! If you're going to hint at assault distributions how accurate was my data

http://mwomercs.com/...ost__p__2220919

Namely:

Lights: 11.1%
Mediums: 16.7%
Heavies: 37.2%
Assaults: 34.4%

(pre assault weekend starting of course)

Edited by hammerreborn, 19 April 2013 - 08:10 PM.

[Hayashi]

Chemie, on 19 April 2013 - 02:29 PM, said:

Also, forcing weight balance in 8-man is crazy because it is so empty. You need different tolerance on 4-man and 8-man queues.

My 8 man has never gotten to drop even once. Just set the ELO tolerance to be good for 4man to PuG only, then completely eliminate ELO for 8 man games. When we're talking coordinated groups, ELO's irrelevant anyway - the results will be far more determined by the way the group works together, over individual skill - so forcing it on 8mans is pointless.

[Dr Killinger]

Dropping with a team of 2 very new players a mate has introduced to the game recently has been hell for me- not sure if it's because of the tourney weekend or what, but we're getting matched against players better than me, who has been playing for 8 months, so the new players have really been getting it bad.

I'll try some solo drops and see what happens. I'd like to know exactly what is done wrt ELO when we're in a diverse group.

As for weight- yes, it's been balanced, but there are seldom fewer than 4 assaults, and have even seen 7 in some matches. That could be the tournament, or the current meta (which sucks, I'm sad to say), so I'll reserve my opinion until after the weekend.

Omid Kiarostami, on 19 April 2013 - 07:38 PM, said:

I can step in and share something on that: We measured a 10% increase in the total number of players matching with Assault-class Mechs, relative to the other weight classes. I don't believe an increase of this nature would cause the weight balance of matches to improve like this, though - if anything, if it were large enough to be significant, it might cause things to get worse by creating more outlying matches (with the other weight classes) that would be harder to balance. It's tough to say.

Wow, I honestly would never have guessed the increase was so low. It's been assault mechs up to the eyes for me in almost all matches. Was the amount of assault mechs already inflated because of the current meta of huge alpha at max range?

Thanks for the update

Edited by Dr Killinger, 19 April 2013 - 11:04 PM.

[Chemie]

Two questions:
1. When you say "balanced" or "grossly not balanced", do you mean weight class or tons? Because you could be 160 tons out with 8x8 assaults; is this 100% balanced by your math?

2. I believe you are getting better weight class balance because there is less diversity in the queue. Have you run TON avarage and standard deviation of the queue (not the game matches). I bet that the average has steadily increase the past month and the std dev has decreased reflecting almost no mediums and fewer lights entering the queue.

[Hauser]

Looking at those graphs I conclude the following:

• The smallest variation between individual bars appears to be slightly less then one pixel. The number of pixels between the height markers is 86 so it would be reasonable to assume that the height markers are placed at the hundreds.
  
• Each elo value appears to be plotted as its own bucket. There are 58 pixels per 100 buckets.
  
• So for every 10.000 players we have 4988 pixels. Or ~2 players per pixel.

I applied a threshold to separate the graphs out and counted the blue pixels.


The total players graph covers ~50k pixels or ~100k players

The graph of players with more then 50 games covers ~25k pixels or ~50k players.

Edited by Hauser, 20 April 2013 - 04:06 AM.