Jump to content

- - - - -

Matchmaking Phase 4 Follow-Up - Feedback


277 replies to this topic

#1 Kyle Polulak

    <member/>

  • Overlord
  • Overlord
  • 584 posts
  • LocationVancouver, BC

Posted 19 April 2013 - 12:47 PM

With the recent news from the following Command Chair Announcement:
http://mwomercs.com/...ost__p__2265319


Spoiler


This thread shall serve as the official response channel.
Please keep your feedback, responses, comments and concerns limited to this thread so that we may both consolidate your responses and ensure no valuable opinions are lost.

Please also remember to keep your responses in line with the Code of Conduct and direct your comments to us (PGI/IGP) rather than arguing with other respondents. Let's keep it constructive!

#2 Lazydrones541

    Member

  • PipPipPipPipPip
  • Overlord
  • Overlord
  • 166 posts

Posted 19 April 2013 - 01:28 PM

So far I must say I am quite pleased with the way the Elo is working now as it no longer seems like a fat kid on a see-saw.
Havent noticed but maybe 1 game in which the teams were horribly unbalanced (we still won - the lighter side.)
Only a few games of noticeably uneven KD matches. Other than that they seem to be reasonably even.
(Most seemed to be within 2-3 left alive on either team; aside from of course cap games)
So far, a good job indeed.

Edited by Lazydrones541, 19 April 2013 - 01:29 PM.


#3 Capt Cole 117

    Member

  • PipPipPipPipPipPip
  • 362 posts
  • LocationSeattle Aerospace Defense Command, Terra

Posted 19 April 2013 - 01:29 PM

Reserved for witty comment.

#4 5th Fedcom Rat

    Member

  • PipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 893 posts

Posted 19 April 2013 - 01:43 PM

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.

#5 WardenWolf

    Member

  • PipPipPipPipPipPipPipPip
  • Legendary Founder
  • 1,684 posts
  • LocationTerra

Posted 19 April 2013 - 02:02 PM

My biggest concern left with matchmaking is the experience of new players. The light blue on the Elo chart is people with less than 50 games - aka new players. They are right there in the middle of the distribution, which means you are dropping new players in with 'average' veteran players... which means about half of those new players (those destined for lower Elo eventually) will struggle and may bail on the game if they have bad experiences early-on (matched against better players + stuck in trial mechs).

I would still suggest having the newer players come in at a lower Elo area, rather than right in the middle. I'd be interested in seeing how that might help the new player experience, and what it might do to the distribution graph as well.

#6 BumRuckus

    Member

  • PipPip
  • The Bold
  • The Bold
  • 25 posts

Posted 19 April 2013 - 02:15 PM

I think that weight balancing is less of an issue than people percieve it to be. I notice among some players I am matched with, when we loose badly they immediately blame it on a "sync-drop" or being "out-tonned." I think this is a fallacy and it's harder for people to step back and say, "we just made bad decisions," or "we were out-played."

To be quite honest, I like the mismatched weight balance in games, it allows for a much greater variety in game play. When the games are too equally matched all the time, they get boring. When I'm in my Raven 2X, I want to play against heavies and assaults sometimes, not always other light-squads.

To me, the most important thing to match is skill. A good ELO matching is going to give people the best games no matter what. A highly skilled heavy team vs a highly skilled lighter team is going to be a good match. Period. This is especially important in 8-mans. A team should be allowed to make their OP ideal composition, be it 8 raven 3Ls or 8 Atlas DDC's, and still be matched against equally skilled players. If everyone wants to brawl in the middle of the map against exactly the same weight team, they might as well flip coins all day on their desktop.

Edited by BumRuckus, 19 April 2013 - 02:17 PM.


#7 armyof1

    Member

  • PipPipPipPipPipPipPipPip
  • Bad Company
  • 1,770 posts

Posted 19 April 2013 - 02:22 PM

View PostWardenWolf, on 19 April 2013 - 02:02 PM, said:

My biggest concern left with matchmaking is the experience of new players. The light blue on the Elo chart is people with less than 50 games - aka new players. They are right there in the middle of the distribution, which means you are dropping new players in with 'average' veteran players... which means about half of those new players (those destined for lower Elo eventually) will struggle and may bail on the game if they have bad experiences early-on (matched against better players + stuck in trial mechs).

I would still suggest having the newer players come in at a lower Elo area, rather than right in the middle. I'd be interested in seeing how that might help the new player experience, and what it might do to the distribution graph as well.


This is exactly what I've been thinking, new players should not be dropped right down into mid-tier players as most of them would be basically punching bags. It's not fun for them and it's not fun for the experienced players to have them in their teams either. Looking at the blue graph I'd say drop them in at around 1050 as they won't have trouble finding matches there and it's opponents that won't blow up their cockpits on a regular basis.

Edited by armyof1, 19 April 2013 - 02:45 PM.


#8 Mystere

    Member

  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Bad Company
  • Bad Company
  • 22,783 posts
  • LocationClassified

Posted 19 April 2013 - 02:23 PM

View Post5th Fedcom Rat, on 19 April 2013 - 01:43 PM, said:

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.


Given that I am really leery of numbers just being thrown around, I'd like to request that you provide more details on the highlighted "facts".

Edited by Mystere, 19 April 2013 - 02:24 PM.


#9 Chemie

    Member

  • PipPipPipPipPipPipPipPipPip
  • Bad Company
  • Bad Company
  • 2,491 posts
  • LocationMI

Posted 19 April 2013 - 02:29 PM

The reason you are getting better weight balance is not the ELO change. It is because everyone and their dog is dropping in highlanders right now so there is less diversity of mech class in the queue. There is no reason that changing ELO would cause better weight balance. You are implying that the same skilled players drop in the same chassis which is stupid.

Also, forcing weight balance in 8-man is crazy because it is so empty. You need different tolerance on 4-man and 8-man queues.

Edited by Chemie, 19 April 2013 - 02:31 PM.


#10 Zylo

    Member

  • PipPipPipPipPipPipPipPip
  • 1,782 posts
  • Locationunknown, possibly drunk

Posted 19 April 2013 - 03:53 PM

PGI needs to look at why matches are "balanced". As others have said before it's probably a result of so many assault mechs in use. PGI should look at the weight class distribution in "balanced" matches. Does each team have nearly all assault mechs in a "balanced" match while a group taking lights + mediums ends up in a very imbalanced match with no lights or mediums on the enemy team?

#11 Glucose

    Member

  • PipPipPipPipPipPip
  • Elite Founder
  • Elite Founder
  • 286 posts

Posted 19 April 2013 - 04:20 PM

I just want to say - great follow up post on the ELO matchmaking. I loved the graphs and the visualization. It's very reassuring that you're taking real measurements and presenting them in this sort of fashion.

I personally, noticed a huge improvement in my teammates skill in the past few days playing. I could tell the matchmaking was working at a higher level.

Thanks for the hard work.

#12 Devil Fox

    Member

  • PipPipPipPipPipPipPipPip
  • The Infernal
  • The Infernal
  • 1,393 posts
  • LocationThe Fox Den

Posted 19 April 2013 - 04:58 PM

View PostZylo, on 19 April 2013 - 03:53 PM, said:

PGI needs to look at why matches are "balanced". As others have said before it's probably a result of so many assault mechs in use. PGI should look at the weight class distribution in "balanced" matches. Does each team have nearly all assault mechs in a "balanced" match while a group taking lights + mediums ends up in a very imbalanced match with no lights or mediums on the enemy team?


That's how it happens... everyone is assault heavy, and their crawling all over the shop in assault mode. So Light's are migrating to conquest... had 7 Cicada and below mechs and one Jaegermech vs 3 assault, 3 Heavy, and 1 each of the fast medium/light... seriously out-tonned but we still killed 5/8 of their team in a open brawl.

The one problem is that I can still be waiting for a little while for a game, now you say the teams are roughly skill balanced but the elo seems to quite happily throw itself open to putting in trial mechs into games, and majority of those continue to disconnect.

#13 Jetfire

    Member

  • PipPipPipPipPipPipPipPip
  • Legendary Founder
  • Legendary Founder
  • 1,746 posts
  • LocationMinneapolis, MN

Posted 19 April 2013 - 05:23 PM

View Post5th Fedcom Rat, on 19 April 2013 - 01:43 PM, said:

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.


Only time will tell, but it certainly didn't hurt the numbers.

#14 Kageru Ikazuchi

    Member

  • PipPipPipPipPipPipPipPip
  • The Determined
  • The Determined
  • 1,190 posts

Posted 19 April 2013 - 05:33 PM

The comparison between the two graphs makes perfect sense to me ... and this is why:
  • I have played about 1000 matches since you started seeding Elo scores (in mid February)
  • since about mid-March, my matches had gotten progressively harder (and my stats kept getting worse)
  • since you re-computed Elo (I'm guessing my scores all went down, except Assault - I almost never play assault) the matches have gotten much easier, and my stats appear to be trending up (I'll know for sure Tuesday, I track them weekly)
So I think I'm "rubber banding" a bit with my Elo scores, but they should settle down in a week or so.

Thanks for the clarity and communications.

#15 Omid Kiarostami

    Member

  • Developer
  • Developer
  • 35 posts

Posted 19 April 2013 - 07:38 PM

View PostMystere, on 19 April 2013 - 02:23 PM, said:


View Post5th Fedcom Rat, on 19 April 2013 - 01:43 PM, said:

The reason your weight balancing improved from 74 to 88% probably has less to do with the new ELO system and everything to do with the fact that three quarters of the people in high ELO brackets are all running highlanders (or other assaults) since the last patch. Correlation does not equal causation.


Given that I am really leery of numbers just being thrown around, I'd like to request that you provide more details on the highlighted "facts".


I like the way you think, Mystere. :D

I can step in and share something on that: We measured a 10% increase in the total number of players matching with Assault-class Mechs, relative to the other weight classes. I don't believe an increase of this nature would cause the weight balance of matches to improve like this, though - if anything, if it were large enough to be significant, it might cause things to get worse by creating more outlying matches (with the other weight classes) that would be harder to balance. It's tough to say.

The point is kind of academic, though; the reason weight balance has improved is entirely different. It's behaving better because the match maker no longer has to stretch across the entire breadth of the Elo curve in order to find matches.

To break it down, what's significant about the new distribution is that there's now something around a 500 point Elo delta containing the majority of players in our system. On the old curve, you can observe a clear 1300-point gulf splitting the player base in two.

This is important if you know that the constraints on Elo and Weight are broadening together with time. So previously, by the time matchmaker had reached across the curve to pick up some more players it had already lowered it's standards on weight significantly. Now that it doesn't need to do this it's finding better matches faster.

So in short: Better data is allowing the match maker to provide better results.

Edit: I posted this before reading Thontor's reply. He summed it up pretty well for me.

Edited by Omid Kiarostami, 19 April 2013 - 07:53 PM.


#16 hammerreborn

    Member

  • PipPipPipPipPipPipPipPipPip
  • Legendary Founder
  • Legendary Founder
  • 3,063 posts
  • LocationAlexandria, VA

Posted 19 April 2013 - 08:08 PM

View PostOmid Kiarostami, on 19 April 2013 - 07:38 PM, said:


I like the way you think, Mystere. :D

I can step in and share something on that: We measured a 10% increase in the total number of players matching with Assault-class Mechs, relative to the other weight classes. I don't believe an increase of this nature would cause the weight balance of matches to improve like this, though - if anything, if it were large enough to be significant, it might cause things to get worse by creating more outlying matches (with the other weight classes) that would be harder to balance. It's tough to say.

The point is kind of academic, though; the reason weight balance has improved is entirely different. It's behaving better because the match maker no longer has to stretch across the entire breadth of the Elo curve in order to find matches.

To break it down, what's significant about the new distribution is that there's now something around a 500 point Elo delta containing the majority of players in our system. On the old curve, you can observe a clear 1300-point gulf splitting the player base in two.

This is important if you know that the constraints on Elo and Weight are broadening together with time. So previously, by the time matchmaker had reached across the curve to pick up some more players it had already lowered it's standards on weight significantly. Now that it doesn't need to do this it's finding better matches faster.

So in short: Better data is allowing the match maker to provide better results.

Edit: I posted this before reading Thontor's reply. He summed it up pretty well for me.


Hey! If you're going to hint at assault distributions how accurate was my data :unsure:

http://mwomercs.com/...ost__p__2220919

Namely:

Lights: 11.1%
Mediums: 16.7%
Heavies: 37.2%
Assaults: 34.4%

(pre assault weekend starting of course)

Edited by hammerreborn, 19 April 2013 - 08:10 PM.


#17 Hayashi

    Snowflake

  • PipPipPipPipPipPipPipPipPip
  • Bridesmaid
  • Bridesmaid
  • 3,395 posts
  • Location輝針城

Posted 19 April 2013 - 08:15 PM

View PostChemie, on 19 April 2013 - 02:29 PM, said:

Also, forcing weight balance in 8-man is crazy because it is so empty. You need different tolerance on 4-man and 8-man queues.

My 8 man has never gotten to drop even once. Just set the ELO tolerance to be good for 4man to PuG only, then completely eliminate ELO for 8 man games. When we're talking coordinated groups, ELO's irrelevant anyway - the results will be far more determined by the way the group works together, over individual skill - so forcing it on 8mans is pointless.

#18 Dr Killinger

    Member

  • PipPipPipPipPipPipPipPip
  • Big Brother
  • Big Brother
  • 1,236 posts
  • LocationJohannesburg, South Africa

Posted 19 April 2013 - 11:00 PM

Dropping with a team of 2 very new players a mate has introduced to the game recently has been hell for me- not sure if it's because of the tourney weekend or what, but we're getting matched against players better than me, who has been playing for 8 months, so the new players have really been getting it bad.

I'll try some solo drops and see what happens. I'd like to know exactly what is done wrt ELO when we're in a diverse group.

As for weight- yes, it's been balanced, but there are seldom fewer than 4 assaults, and have even seen 7 in some matches. That could be the tournament, or the current meta (which sucks, I'm sad to say), so I'll reserve my opinion until after the weekend.


View PostOmid Kiarostami, on 19 April 2013 - 07:38 PM, said:

I can step in and share something on that: We measured a 10% increase in the total number of players matching with Assault-class Mechs, relative to the other weight classes. I don't believe an increase of this nature would cause the weight balance of matches to improve like this, though - if anything, if it were large enough to be significant, it might cause things to get worse by creating more outlying matches (with the other weight classes) that would be harder to balance. It's tough to say.


Wow, I honestly would never have guessed the increase was so low. It's been assault mechs up to the eyes for me in almost all matches. Was the amount of assault mechs already inflated because of the current meta of huge alpha at max range?

Thanks for the update :lol:

Edited by Dr Killinger, 19 April 2013 - 11:04 PM.


#19 Chemie

    Member

  • PipPipPipPipPipPipPipPipPip
  • Bad Company
  • Bad Company
  • 2,491 posts
  • LocationMI

Posted 20 April 2013 - 03:54 AM

Two questions:
1. When you say "balanced" or "grossly not balanced", do you mean weight class or tons? Because you could be 160 tons out with 8x8 assaults; is this 100% balanced by your math?

2. I believe you are getting better weight class balance because there is less diversity in the queue. Have you run TON avarage and standard deviation of the queue (not the game matches). I bet that the average has steadily increase the past month and the std dev has decreased reflecting almost no mediums and fewer lights entering the queue.

#20 Hauser

    Member

  • PipPipPipPipPipPipPip
  • Legendary Founder
  • Legendary Founder
  • 976 posts

Posted 20 April 2013 - 03:59 AM

Looking at those graphs I conclude the following:
  • The smallest variation between individual bars appears to be slightly less then one pixel. The number of pixels between the height markers is 86 so it would be reasonable to assume that the height markers are placed at the hundreds.
  • Each elo value appears to be plotted as its own bucket. There are 58 pixels per 100 buckets.
  • So for every 10.000 players we have 4988 pixels. Or ~2 players per pixel.
I applied a threshold to separate the graphs out and counted the blue pixels.


The total players graph covers ~50k pixels or ~100k players

The graph of players with more then 50 games covers ~25k pixels or ~50k players.

Edited by Hauser, 20 April 2013 - 04:06 AM.






4 user(s) are reading this topic

0 members, 4 guests, 0 anonymous users