1911 replies to this topic

[Karl Berg]

SgtKinCaiD, on 01 July 2014 - 05:49 AM, said:

@Karl

Thanks for your time and response, but i got another question :



So, from what i understand from your response, the ELO bucket are not re-introduced with the 4*3. Now i would like to know if the new MM will still use the rule of balancing a high ELO and a low ELO player against 2 medium ELO players ?
Matthew Craig post for reference : http://mwomercs.com/...ost__p__2633230

Bonus question : did you tune the ELO scoring formula for the new MM so that it's not based on Win/Loss only ?

The matchmaker doesn't specifically set out with this goal, no. It starts by seeding with a relatively old request, and uses the calculated Elo of that request to iteratively search the matchmaking queue for similarly skilled team members and opponents. Over time, the window of requests that it examines widens. That said, if one team ends up significantly ahead of the other team in terms of Elo, we have a tunable blending formula that can bias the search in an attempt to re-balance the teams.

[Karl Berg]

Modo44, on 02 July 2014 - 10:11 PM, said:

Now that the matchmaker makes sense, there are two obvious improvements:

Track group and solo Elo separately. You keep talking about adding a set modifier, but that has obvious issues. You can not predict who sucks the same (or even more) when grouped up. Pushing group Elo manually would introduce new problems, specifically: unduly punishing bad players by asking them to carry unrealistically hard. Tracking group Elo would automatically adjust for every player's group play (in)ability.

Track chassis Elo separately. Seeding issues were mentioned for this. One solution would be using MWO-specific battle values (i.e. averaged Elo values for each chassis -- already tracked) to help with initial seeding and re-seeding of chassis Elo. Again, no manual adjustments, just working with heuristics based on Elo.

We are thinking along very similar lines.. We're currently in the middle or data-mining and experimenting with various tracking metrics and adjustment algorithms to determine how best we should be modifying Elo to suit the game. The data set is so large that I have to run the experiments on a 64-bit process, due to the enormous amounts of memory it requires.

We're evaluating based on many measures, including how rapidly the algorithm converges to the predicted Elo curve compared to measured team deltas, how much aggregate error remains in that curve, how accurately it predicts win/loss, etc..

As for group-size set modifiers, I have some pretty hard numbers on win-percent increase based on group size dynamics. Unfortunately I got these numbers too late to do anything for the version of the matchmaker that is on production right now.

The bias based on group size is highly statistically significant. So much so, that even adding in these simple biases should have a highly noticeable effect in the 'fairness' of the group queue. Right now the matchmaker is essentially blind to group size based win percentage impacts, and that's something I want to address as soon as I possibly can. The biases are run-time tunable of course; so until we can fix per-user Elo metrics, we can continue to adjust these group size modifiers to accurately reflect production behaviours.

[Modo44]

Karl Berg, on 05 July 2014 - 11:57 AM, said:

The bias based on group size is highly statistically significant. So much so, that even adding in these simple biases should have a highly noticeable effect in the 'fairness' of the group queue. Right now the matchmaker is essentially blind to group size based win percentage impacts, and that's something I want to address as soon as I possibly can. The biases are run-time tunable of course; so until we can fix per-user Elo metrics, we can continue to adjust these group size modifiers to accurately reflect production behaviours.

If that is easy to implement, it sounds like a great idea. The only worry would be breaking things for big casual groups, but hopefully that can be tracked, too. (Given the amount of carrying seen on competitive player's streams, I would not be worried about giving them a harder time. Some people seem bored after the patch. )

[Karl Berg]

Modo44, on 05 July 2014 - 12:12 PM, said:

If that is easy to implement, it sounds like a great idea. The only worry would be breaking things for big casual groups, but hopefully that can be tracked, too. (Given the amount of carrying seen on competitive player's streams, I would not be worried about giving them a harder time. Some people seem bored after the patch. )

It's also worth noting that for small groups, the values we're talking about are not extreme. For example, a group of 2 has a 2.4734% +-0.596% chance increase of winning against a team of solo players with a 99% confidence interval. That works out to a mere 19 point Elo adjustment for a group of 2 to compensate for that bias.

This increases non-linearly with group size, and I have much poorer data for 5+ groups due to the reduced set of sample data. But as we continue gathering data on production I will continue to be able to refine these numbers and improve these adjustments.

[Goose]

So:

• We are moving from "Elo" to "Karl Berg?" Outstanding.
  
• Group size can be a flat modifier, thus you "only" need to double the Elo KB count to "weight class * group (yes/ no)"?
  
• Doesn't sound like there's a push to change KB to per chassis instead of per weight class. Shirley that would be the way to illustrate the significance of build …
  
• (This is one too far by my own standards, but it needs to be asked) Is there any regard for game mode in all of this?

[Kageru Ikazuchi]

Karl Berg, on 05 July 2014 - 12:44 PM, said:

It's also worth noting that for small groups, the values we're talking about are not extreme. For example, a group of 2 has a 2.4734% +-0.596% chance increase of winning against a team of solo players with a 99% confidence interval. That works out to a mere 19 point Elo adjustment for a group of 2 to compensate for that bias.

This increases non-linearly with group size, and I have much poorer data for 5+ groups due to the reduced set of sample data. But as we continue gathering data on production I will continue to be able to refine these numbers and improve these adjustments.

http://youtu.be/7DKIS1oXRdU?t=37m5s
Karl, if you have not watched this discussion about the group queue, you should ... ROFLWaffle hosts the War Room each Thursday night, and it could be a great venue to get some of these tuning features and tweaks out into the open.

[Deathlike]

Karl, could you please consider this idea?

Copied and pasted from a different thread:

Quote

TBH, I wish it was a tiered system as I've described before...

1) Solo queue - as in true "solo" only
2) Small group queue - 2-4 man premades, with opt-in solos
3) Big group queue - 5 to 12-man groups, with opt-in solos and opt-in small groups

Ideally, people would opt in as they get better... possibly provided with a reward like +10% XP and/or +10% C-bill growth (though, I lean towards C-bills because XP gains are pretty easy).

The idea is that you move slowly up "ladder-wise" with your Elo, instead of jumping off in the deep end with the big boys.

My suggestion also has some contingencies, specifically "rewards" for people opting in. It doesn't necessarily have to be a flat 10%... in fact it should scale to like a 20 to 25% C-bills bonus, as the queues need.... including the Solo queue's Light+Medium starvation points.

While it won't fix skill levels, it will sort people out better and provide them a reason (not necessarily a good one, but whatever it takes) to make the MM's job easier, while not hurt people that are trying to learn the game incrementally.

[Imperius]

So with my new system up and running I'm now working to tweak the game to look it's best while staying 50-60 FPS. I think I found something you guys/girls may need to tweak when you get around to an optimization run.

While I was testing some settings in "Testing Grounds" I noticed and issue that came up with lasers and firing up close.

Map: Crimson Strait
Issue: FPS drop of 20-32 FPS when firing lasers up close into terrain and water.
How to replicate: Fire lasers into terrain/water up-close.

Notes: If you fire the lasers at terrain/water at afar no FPS issue. I think this is a issue cause by the amount or way the damage effect sprites of water/terrain combined with light are rendered to user up close.

System settings

Very High
Post-AA
V-Sync ON
Damage Glow ON

Tested in a private match with friends one with low settings the other High no tweaks

User High Settings: Same issue
User Low Settings: No issue

Custom settings and DxDiag

Spoiler

Edited by Imperius, 06 July 2014 - 02:22 AM.

[Reno Blade]

Imperius, on 06 July 2014 - 02:17 AM, said:

So with my new system up and running I'm now working to tweak the game to look it's best while staying 50-60 FPS. I think I found something you guys/girls may need to tweak when you get around to an optimization run.

While I was testing some settings in "Testing Grounds" I noticed and issue that came up with lasers and firing up close.

Map: Crimson Strait
Issue: FPS drop of 20-32 FPS when firing lasers up close into terrain and water.
How to replicate: Fire lasers into terrain/water up-close.

Notes: If you fire the lasers at terrain/water at afar no FPS issue. I think this is a issue cause by the amount or way the damage effect sprites of water/terrain combined with light are rendered to user up close.

System settings

Very High
Post-AA
V-Sync ON
Damage Glow ON

Tested in a private match with friends one with low settings the other High no tweaks

User High Settings: Same issue
User Low Settings: No issue

Take a flamer and test this again. It could even be worse.
The smoke and impact effects up close (less than 30m) is enourmous.
Going down from 50 to 10 fps that way.
Gladly, it's less stress hitting a mech with a Flamer, but the smoke and steam on cold maps is another issue that can only be reduced by setting your smoke settings (Effects) to low.

[Imperius]

I didn't go test every weapon, but the issue is there, and most likely a simple fix and PGI may not be aware of the issue.

[Goose]

I believe you should be having a discussion of settings up in Hardware …

[Cimarb]

Deathlike, on 06 July 2014 - 12:25 AM, said:

Karl, could you please consider this idea?

Copied and pasted from a different thread:
_____________________________________________
TBH, I wish it was a tiered system as I've described before...

1) Solo queue - as in true "solo" only
2) Small group queue - 2-4 man premades, with opt-in solos
3) Big group queue - 5 to 12-man groups, with opt-in solos and opt-in small groups

Ideally, people would opt in as they get better... possibly provided with a reward like +10% XP and/or +10% C-bill growth (though, I lean towards C-bills because XP gains are pretty easy).

The idea is that you move slowly up "ladder-wise" with your Elo, instead of jumping off in the deep end with the big boys.
______________________________________________

My suggestion also has some contingencies, specifically "rewards" for people opting in. It doesn't necessarily have to be a flat 10%... in fact it should scale to like a 20 to 25% C-bills bonus, as the queues need.... including the Solo queue's Light+Medium starvation points.

While it won't fix skill levels, it will sort people out better and provide them a reason (not necessarily a good one, but whatever it takes) to make the MM's job easier, while not hurt people that are trying to learn the game incrementally.

I fully support that, and it is actually a better system than the simple "solo opt-in" I was suggesting.

[Imperius]

Goose, on 06 July 2014 - 01:58 PM, said:

I believe you should be having a discussion of settings up in Hardware …

No has nothing to do with hardware... Strictly an optimization issue. Karl Burg is the man and he gets stuff done/fixed

[Helmer]

Karl Fanboi:


Niko tweets "Monday Morning Coffee and a new Webcam "

You think.... "Hey there's Karl! Wonder how he's doing"





Cheers.

Edited by Helmer, 07 July 2014 - 12:57 PM.

[Cimarb]

Karl, Sasparilla Kid shared a screenshot and proved me wrong in my defense of the MM:

Sarsaparilla Kid, on 06 July 2014 - 10:41 PM, said:

Here's mine...we were a 12-man, and obviously the other side was at least 2 separate groups matched up against us. We had 3/3/3/3, but they had 4/4/2/2:

His 12-man went against at least two smaller groups, and both assault and heavy valves were released, giving the smaller groups a 2/2/4/4 spread compared to the 12-man 3/3/3/3 they are forced to use. I assume that is an extreme situation and MM just couldn't find them a match otherwise, but is that intentional? I am fine with valves if they still maintain the balance between teams, but a situation like this should not happen by what I have read.

(Edited to add SK's quote)

Edited by Cimarb, 07 July 2014 - 01:14 PM.

[East Indy]

Goose, on 05 July 2014 - 12:44 PM, said:

We are moving from "Elo" to "Karl Berg?"

"KARLO"

[Wintersdark]

Cimarb, on 02 July 2014 - 10:37 AM, said:

That would be awesome if you could have it toggle DPI setting when you zoom in... Not sure if that would be as feasible as doing it manually when you need it, but...

I have my mouse set up to do this. Mouse thumb button sends max zoom keybind and drops my DPI 1500 points. Release to set minimum zoom keybind and return DPI to normal.

[Wintersdark]

Cimarb, on 07 July 2014 - 01:13 PM, said:

Karl, Sasparilla Kid shared a screenshot and proved me wrong in my defense of the MM:

His 12-man went against at least two smaller groups, and both assault and heavy valves were released, giving the smaller groups a 2/2/4/4 spread compared to the 12-man 3/3/3/3 they are forced to use. I assume that is an extreme situation and MM just couldn't find them a match otherwise, but is that intentional? I am fine with valves if they still maintain the balance between teams, but a situation like this should not happen by what I have read.

(Edited to add SK's quote)

This seems entirely reasonable to me. There's a noted bias in favour of larger teams in match results, as Karl has noted earlier. One downside of large(well, all) premade groups is that you must follow the 3/3/3/3 rules. A team made up of smaller premade groups may then break this, though each group in and of itself follows the 3-max rules.

Its an unavoidable consequence of the system and numbers of groups searching at any given time.

Note that the end result was still very close - Just +1/+1/-1/-1; not like 10 Assaults and 2 Heavies (we've all seen those matches in the past!)

The solo queue works much better for class matching, for obvious reasons.

[Cimarb]

Wintersdark, on 07 July 2014 - 02:28 PM, said:

This seems entirely reasonable to me. There's a noted bias in favour of larger teams in match results, as Karl has noted earlier. One downside of large(well, all) premade groups is that you must follow the 3/3/3/3 rules. A team made up of smaller premade groups may then break this, though each group in and of itself follows the 3-max rules.

Its an unavoidable consequence of the system and numbers of groups searching at any given time.

Note that the end result was still very close - Just +1/+1/-1/-1; not like 10 Assaults and 2 Heavies (we've all seen those matches in the past!)

The solo queue works much better for class matching, for obvious reasons.

I am not saying it is unreasonable, but the major advantage of the new MM is that it ALWAYS makes sure that the teams are equal weight classes. Aside from that, and 5+ groups, the new MM has not added anything. Those are two HUGE improvements, granted.

The end result of Sarsaprilla's example was still very close, but if you go to that thread, it spawned because another person was complaining that there were 6 Assaults on the enemy team, to their 12-man's 3 Assaults. That person did not have any proof of this, but the fact that it IS possible is troubling.

Basically, I do not want it possible for 3 groups of 4 to have 3 Assaults and a Heavy, vs. a 12-man that, communication or not, would be at a drastic disadvantage due to the weight disparity. I would prefer it to fail to make the match, personally.

[Wintersdark]

Cimarb, on 07 July 2014 - 04:02 PM, said:

Basically, I do not want it possible for 3 groups of 4 to have 3 Assaults and a Heavy, vs. a 12-man that, communication or not, would be at a drastic disadvantage due to the weight disparity. I would prefer it to fail to make the match, personally.

This is entirely a tuning issues. Whether it should fail or not, well, that's really a ball in PGI's court.

I'm really not concerned either way - I've never been bothered by moderate tonnage mismatches, but I can respect differing opinions there.

Personally, I think they should have groups limited to a hard 3/3/3/3, and matches overall limited to a hard 3/3/3/3 +/- 1 per class, so the heaviest a team can run is 4/4/2/2, and only when it's comprised of multiple small groups.

But... *shrugs* like I said, I'm not terribly concerned either way. I rarely play in 12 mans, so the reality of 12-man matchmaking searches going back to failing multiple times doesn't impact me so much.

I'd assume the 12-mans would prefer to actually reliably get matches, but I could well be wrong.

Edited by Wintersdark, 07 July 2014 - 04:15 PM.