1911 replies to this topic

[Wintersdark]

Cimarb, on 27 May 2014 - 04:26 PM, said:

"Elo is thought of as pilot skill, but it is not pilot skill. Elo is a measure of how much YOU contribute to victory on average." (caps mine)
- True, but Elo doesn't show that. It shows how good or poor your luck with groups has been, on average, across your lifetime. Being hated by the RNG Gods, I can tell you that my luck is HORRIBLE. I can contribute in every way possible, but if I get teamed up with horrible teams over and over and over again, my Elo is going to be abominable despite everything I contributed (or not).

It is indeed imperfect, but it is also the best option available for all the reasons I've given above. There simply isn't a better, viable option.

Quote

It has nothing to do with my skill, nor my contribution. If I am a member of a 4-man team for my first six months and sit on the spawn point while they win over and over and over again, despite my complete lack of contribution, my Elo will be exactly the same as theirs. That is something that is happening all over the place currently, btw (just not to that extreme).

It has everything to do with your contribution. Your specific contribution may well be zero, but that will impact the likelyhood of your team's winning the match. The hard reality is that even in the example you give, you'll end up with the correct Elo.

Remember - pay close attention here - Elo is not your skill level. Elo represents your likelyhood of winning matches, in the manner in which you play most frequently. If you always play in a 4-man, and do nothing but still win it's because the three with you are very good - however, they're not as good as 4 players equal to the three carrying you. Regardless, the 4 of you together (which is how you most commonly play) end up with the correct Elo value, representative of your group.

This will cause funny mismatches in random pugs, where you'll be matched against teams with higher Elo values, but that will be in the minority of matches for you, simply because if you start playing a lot of matches solo, your Elo value will quickly adapt to match your new circumstances.

Quote

As far as BV, it is far less complex than 98% of what Karl does. BV is simply assigning a value to the combination of weapons and equipment on the mech. Since all mechs use the same equipment in different quantities, even though there are immense numbers of combinations, it is a very simple calculation once you get initial values, and it would only need changed when a weapon or piece of equipment was changed, or for fine-tuning balance (if that was even needed).

Now, I don't mean to be offensive here, but this statement shows a ridiculous lack of knowledge about game design. Assigning points values to disparate items to represent their value is extremely difficult. Even in tabletop wargaming where game mechanics are rather static and highly predictable mathematically, it's extremely challenging to get right.

Quote

You could use TT values to begin with, actually, since the stats for weapons are relatively the same. You could do some tinkering to take into account FLD and other MWO-only issues, but those would be minor factors.

As above, this is not so simple. You're totally glossing over how hard it is to get the correct values. Remember, if your values are incorrect, you basically cause what's equivilant to an Elo mismatch except that it is not self-correcting over time, it's permanent. Weapon stats are a part of that valuation, but a small one, and those "minor factors"? They're huge. Massive. And extremely difficult to quanitify. Exactly how much better, in BV, is a PPC vs. a ERLL? You've got range considerations, heat, FLD vs. DOT, ToT, ability to use while poptarting effectively, projectile speed.... Then you have synergy, which is a huge kettle of fish.

Arriving at BV values in Tabletop is comparatively simple. This is because unlike MWO, Tabletop is entirely mathematically predictable. You can build a simulation program to run multiple engagements, and use those results to adjust point values - I used to do this all the time in tabletop wargaming, to assess troop value vs. point cost.

In MWO, you can't predict combat results in the same manner. Groups of weapons can be greater or less than the sum of their parts due to complex mechanics like Ghost Heat, and even basic game mechanics like projectile speed.

Quote

Much like the heat scale, using the system that has been used for decades is a good start and should be used as a benchmark for any system built off of it. Only when that system has been tried and found unusable should a completely new system be attempted.

It hasn't been used for decades. Tabletop is an entirely different game, and the game mechanics are utterly different. Even though basic mech construction and weapons use similar numbers, the application of those basic values are entirely different. The heat system is totally different, aiming is non-random, hardpoint location is critical in ways that are irrelevant in tabletop, the list goes on and on. It's just not a compatible system.

Quote

What you mention about "manipulable[size=4] by enterprising players" is not a fair argument, as you can currently do the exact same thing with Elo - just go into a match and disconnect or, even worse, TK a team member or two, and you have now lowered your Elo. Do that often enough (but not often enough to be obvious), and you can keep your Elo in the bottom bucket quite easily. I'm not sure why you would do that, but same thing as what you are saying people will do with BV.

No, it's not the same thing.

In order to "smurf" your way into lower Elo, you'd basically need to lose the majority of your matches, and continue to do that permenantly. While you could use that potentially to try to abuse some sorts of tournaments (the old-school kind MWO runs, not the private match types), that's about it... And that abuse is extremely counterproductive to you as a player as you still need to lose the majority of your matches permenantly.

With a BV+"Pilot Skill" (based on stats) system, here's what happens.

Say valuation based on stats relative to weapon valuation is incorrect (highly likely). Now players are being matched against foes that are always either too potent or too weak. This is manipulateable in an entirely profitable way.

For example, lets say that Large Lasers are ranked fairly low vs. PPC's, but weapons overall are ranked somewhat high vs. stats. Now, I can equip Large Lasers, and be matched against much poorer players predictably, killing them with impunity while actually doing little damage. My teams win more often, but my stats don't noticably improve - I make a point of crippling enemy mechs with minimal damage done, and I'm facing poor players who don't understand this tactic/can't exploit it effectively. Or, lets say PPC's are ranked too highly, pilot skill to low. Now, poor players equip a PPC, and are suddenly put up against much more skilled opponents, and regularly butchered, but they don't know why. The key here is that players can learn to build mechs or work their stats to deliberately change their resultant battlevalue.

That doesn't happen in Elo. Sure, it can for a few matches when things change, but the system adapts. The developers don't need to come up with values and adjust to balance them; the system is self-balancing over time.

The problem with your argument here is you're totally glossing over the difficult part. In a perfect world where everything could be magically quantified, a Battle Value+Pilot Skill Ranking System would absolutely be a great system. If the numbers, ALL the numbers, are not correct the system fails every time, instead of just when things change. It doesn't self adjust.

Thus it relies on constant, ongoing tuning (note how well that works for weapon stats alone), that's immensely complex as it needs to be adjusted constantly in a volatile unpredictable ecosystem. It's totally unlike tabletop, and absolutely NOT simple.

And still, even if the system is perfect? IT STILL FAILS IN THE SAME CASES AS ELO as per my first post. You'd get matches with disparate battlevalues simply because of the poor availability of players in a given bracket.

Edited by Wintersdark, 27 May 2014 - 05:48 PM.

[Lindonius]

Heffay, on 27 May 2014 - 10:48 AM, said:

Dear Mr Berg,

Recently you stated that the game creates "several tens of thousands" of matches a day. There is some debate on another forum that you are referring to person-matches, so 1 match really means 24 people in that number you referenced (i.e. divide that range by 24 to get the actual number of matches you spawn a day). Can you clarify if you meant matches, or person-matches when you gave that number? Thank you!

Love,
Heffay

Well if you take the opposite as true, (that each match = 24 players), using a ball bark figure of 30,000 that would mean that they are spawning seven hundred and twenty THOUSAND people matches per day.

Which do you think is true, given the current state of the game?

Edited by Lindonius, 27 May 2014 - 06:27 PM.

[p4r4g0n]

Wintersdark, on 27 May 2014 - 05:44 PM, said:

-snip-

Another point to note against the BV system is that whenever any changes are made to the mechs / weapons / equipment, many BV values may have to be re-calculated or at least re-visited to ensure that the values are appropriate. In an on-going development, that's going to be a freaking nightmare.

[Karl Berg]

Heffay, on 27 May 2014 - 10:48 AM, said:

Dear Mr Berg,

Recently you stated that the game creates "several tens of thousands" of matches a day. There is some debate on another forum that you are referring to person-matches, so 1 match really means 24 people in that number you referenced (i.e. divide that range by 24 to get the actual number of matches you spawn a day). Can you clarify if you meant matches, or person-matches when you gave that number? Thank you!

Love,
Heffay

Hey sir,

That stat was for full 24 player games.

As for the matchmaking concerns, it will take me some time to properly read through all these posts to respond. Briefly though; Elo provides a tool to predict match outcome based on past outcomes. If I were to attempt to bias Elo with specific statistical factors, like loadout specific Elo handicaps, it would very likely worsen the systems performance.

Sorry to be so brief; I will try to go into better detail when I'm not quite so time constrained.

[Tekadept]

Hi Karl, just wondering if you have a stat on what is the average number of matches a player plays in a day?, ie 5 games a day, 10 games a day? And what is the biggest outlier, ie some poor soul doing 100 matches a day

[Heffay]

Tekadept, on 27 May 2014 - 07:38 PM, said:

And what is the biggest outlier, ie some poor soul doing 100 matches a day

I think you can just ask Voidsinger for that number.

[Lindonius]

Tekadept, on 27 May 2014 - 07:38 PM, said:

Hi Karl, just wondering if you have a stat on what is the average number of matches a player plays in a day?, ie 5 games a day, 10 games a day? And what is the biggest outlier, ie some poor soul doing 100 matches a day

If the average player plays 10 matches a day, then according to Karl's figures and a 30,000 match a day ball park figure, that makes 72,000 players logging in per day.

[Cimarb]

Lindonius, on 27 May 2014 - 07:51 PM, said:

If the average player plays 10 matches a day, then according to Karl's figures and a 30,000 match a day ball park figure, that makes 72,000 players logging in per day.

and that kind of calculation is why they try not to give out those numbers

[Heffay]

Lindonius, on 27 May 2014 - 07:51 PM, said:

If the average player plays 10 matches a day, then according to Karl's figures and a 30,000 match a day ball park figure, that makes 72,000 players logging in per day.

I really doubt the average number of matches per day is anywhere near 10. My guess is it would be in the 3-5 range. There are a TON of gamers out there with families who try to get a match in or two once or twice a week, and that will dramatically skew the average down.

[Wintersdark]

p4r4g0n, on 27 May 2014 - 06:40 PM, said:

Another point to note against the BV system is that whenever any changes are made to the mechs / weapons / equipment, many BV values may have to be re-calculated or at least re-visited to ensure that the values are appropriate. In an on-going development, that's going to be a freaking nightmare.

This is exactly what I meant by a changing ecosystem.

Its worse than that, though. Not just when stats change... Though consider the impact even a seemingly small stat change had when they initially increased LRM velocity! (Incidentally a stat which doesn't even exist in tabletop and as such isn't considered in base BV numbers)....


But consider, what about maps? If more maps like Alpine are added, the value of long range weaponry such as LRM's, plc's and the like grows. Add more urban environments and otherwise cover-rich maps(+fix SRM's) and their value changes too.

Now try to quantify player skill via statistics (see my earlier post as well with this).... How? How do you get a number you can work into BV to be useful at all? If you could come up with that algorithm you could become a very rich man.

[TheCaptainJZ]

Wintersdark, on 27 May 2014 - 08:05 PM, said:

...lots of excellent explainations...

I just want to say what an amazing job you are doing at explaining why ELO is the only viable matchmaking solution. Good stuff!

[Mawai]

Heffay, on 27 May 2014 - 07:57 PM, said:

I really doubt the average number of matches per day is anywhere near 10. My guess is it would be in the 3-5 range. There are a TON of gamers out there with families who try to get a match in or two once or twice a week, and that will dramatically skew the average down.

It would also change the number of players logging in every day way up since it is the number of full matches that was fixed. If the average number of games played for each player goes down then the number of different players playing goes up to compensate since it is the number of matches (10's of thousands) that is known.

[Mawai]

Wintersdark, on 27 May 2014 - 08:05 PM, said:

This is exactly what I meant by a changing ecosystem.

Its worse than that, though. Not just when stats change... Though consider the impact even a seemingly small stat change had when they initially increased LRM velocity! (Incidentally a stat which doesn't even exist in tabletop and as such isn't considered in base BV numbers)....


But consider, what about maps? If more maps like Alpine are added, the value of long range weaponry such as LRM's, plc's and the like grows. Add more urban environments and otherwise cover-rich maps(+fix SRM's) and their value changes too.

Now try to quantify player skill via statistics (see my earlier post as well with this).... How? How do you get a number you can work into BV to be useful at all? If you could come up with that algorithm you could become a very rich man.

I completely agree that perfection of balance through BV is impossible.

However, that isn't the question. In my opinion, the question is, could a better system of matching be developed than is currently used?

Right now, ELO appears to be the ONLY constraint on the matchmaker. I played a few games yesterday and in three of the matches there were 5 light mechs on my side vs 1 or 2 on the opposing team. In another match or two, there were 4 on my team. Most of those matches did not go well because we were at a significant tonnage disadvantage though a couple of the games were decent matches and we actually won one (possibly two) despite the disadvantage.

If ELO itself was sufficient for good balanced matches then we would not be going through the matchmaker re-write to include weight class/tonnage matching and 3/3/3/3 distributions. What does weight class/tonnage matching represent? It is a very simplified BV system using only the mech tonnage or weight class to distinguish effectiveness. By including equal numbers of each weight class or tonnage on each team ... the idea is that you match mech effectiveness. Match mech effectiveness as well as pilot effectiveness (ELO is a measure of long term performance) and in theory you generate more balanced matches.

So ... what changes could be made that might improve things?

1) Instead of using strict tonnage to match mech effectiveness you could develop an automated system that assesses speed/engine type/heat efficiency/dps and alpha damage/tonnage/durability to estimate a mech's effectiveness. It does not need to be perfect. There would be some inconsistencies ... however, would such a system do a better job than just taking the weight class (which groups jenners and locusts together or cicadas and shadow hawks), or strict tonnage which matches a jenner with 6 flamers with one with 6ML. Each mech would have a base value based on tonnage (since that is what the matchmaker is currently planned to do) ... modified somewhat for player equipment and module choices.

2) The value calculated in 1 for the mech itself could then be modified by a factor based on ELO.

3) The value from 2 ... could then be modified by a factor for being part of a group (alternatively the game could maintain grouped and solo ELO separately and avoid that issue). [The suggestion was made early on in the development process to maintain a grouped and solo ELO for each player so that the effect of groups would be more easily factored into the matchmaking balance ... as far as I know that was never used]

This end value calculated for each mech entering the queue could then be used by the matchmaker to try to balance a match ... the objective being to try to achieve the same total value on each side.

ELO would be adjusted in the same way it is now.

Some examples ... a good jenner pilot with a decent loadout might have the same value as a hunchback with an average pilot or a shadowhawk with a below average pilot.

Advantages:
- the matchmaker has to optimize on one match metric. Building a matchmaker that optimizes multiple metrics simultaneously is hard unless you impose fixed numbers like 3/3/3/3 .. which may cause other issues.
- Pilot skill and mech effectiveness are both factored into the balance. (Keep in mind that they are already planning to use weight class/tonnage as a constraint because ELO by itself does not seem to be sufficient).
- perfection of balance is neither possible nor expected ... we are only looking for something better

Disadvantages:
- it will require some tuning - especially in developing the base mech effectiveness value. These values don't need to be exact ... only ballpark ... the idea is to come up with some estimate that is somewhat better than just weight class or tonnage
- no guarantee that it will actually work (the goal of the effort is to improve matchmaking by including factors for mech effectiveness, pilot effectiveness and teamwork ... however, these may not be the only significant factors and it is possible that match outcomes are more strongly influenced by random factors ... players having to leave for real life reasons going afk or disconnecting, player state of inebriation , casual team vs. serious team)

However, the goal is something that works BETTER ... not something that is perfect ... and since ELO by itself appears to have been demonstrated as insufficient ... then additional factors like mech or team effectiveness need to be factored in to match balancing ... and something other than weight class/tonnage might be a more accurate indicator of this.

Edited by Mawai, 28 May 2014 - 06:35 AM.

[Heffay]

Mawai, on 28 May 2014 - 06:21 AM, said:

However, the goal is something that works BETTER ... not something that is perfect ... and since ELO by itself appears to have been demonstrated as insufficient ... then additional factors like mech or team effectiveness need to be factored in to match balancing ... and something other than weight class/tonnage might be a more accurate indicator of this.

3/3/3/3 + Elo is better than just Elo. The 3/3/3/3 isn't about simulating BV; it helps ensure diversity on the battlefield and actually promotes role warfare. Where are their assaults? Gotta find them, send in your fast attacks from the flanks, figure out where their fast movers are, etc. It creates much more dynamic play than "grab 8-10 heavies and assaults and turtle up."

[Tekadept]

Heffay, on 28 May 2014 - 06:44 AM, said:

3/3/3/3 + Elo is better than just Elo. The 3/3/3/3 isn't about simulating BV; it helps ensure diversity on the battlefield and actually promotes role warfare. Where are their assaults? Gotta find them, send in your fast attacks from the flanks, figure out where their fast movers are, etc. It creates much more dynamic play than "grab 8-10 heavies and assaults and turtle up."

How varied is the roles though when at least 6 of those mechs are touting 2 ppc and an ac10 that end up sniping all match.

You need different Rewards to make the play dynamic and stop turtling up more then weight restrictions. you need the incentive to go out and scout.

Edited by Tekadept, 28 May 2014 - 06:47 AM.

[Heffay]

Tekadept, on 28 May 2014 - 06:46 AM, said:

How varied is the roles though when at least 6 of those mechs are touting 2 ppc and an ac10 that end up sniping all match.

You need different Rewards to make the play dynamic and stop turtling up more then weight restrictions. you need the incentive to go out and scout.

Yah? That's all true too. Make SRMs something to be feared. You'll get fast attack SRM boats that use their speed to get into the lines and out again while causing damage. Heavies and assaults will have to sacrifice some of their sniping weapons to carry their own to defend against them.

That is why getting SRM hit detection fixed is so important, and one of their top priorities. It'll completely change the pace of the game and along with 3/3/3/3 create matches that are even MORE fun!

[Cimarb]

Heffay, on 28 May 2014 - 06:44 AM, said:

3/3/3/3 + Elo is better than just Elo. The 3/3/3/3 isn't about simulating BV; it helps ensure diversity on the battlefield and actually promotes role warfare. Where are their assaults? Gotta find them, send in your fast attacks from the flanks, figure out where their fast movers are, etc. It creates much more dynamic play than "grab 8-10 heavies and assaults and turtle up."

You missed his point, Hefs. 3/3/3/3 is a (very) simplified BV system, just like the weight classifications themselves are. They are all designed to put mechs into buckets, it just depends on how big the bucket is.

There are tons of ways to do BV as well, from Roland's popularity value to TTs pure stat value and everything in between. 3/3/3/3 tries to do it in a very simple manner, by assigning one of four "BV" values (light, medium, heavy, assault) and then restricting how many of each is used. Tonnage restrictions do the same thing, also in a simple manner, by assigning "BV" values in a much more fluid fashion (20 tons to 100 tons, as a portion of the total weight). The more accurate the BV, the more accurate the matches will be (exclusive of skill), so having a BV that is determined by popularity or stats is going to be the most accurate method, because it uses the most specific data.

BV does not have to be perfect, nor even particularly close. All it has to do is give a more accurate number to gauge the abilities of the mech (skill excluded) than gross tonnage.

[TheCaptainJZ]

Yeah, it's true we need something more than ELO but what else to use is the question. I personally don't like the strict 3/3/3/3 rule only because it makes the enemy team composition predictable and less realistic and for me that's less fun. Of course, a team of all lights against a team of all assaults is not going to be fun either. I'd rather have a "loose" 3/3/3/3 rule where the MM can pick a mech on the border of classes or a neighboring class. For instance, a Commando, Raven, Cicada, Centurion, Griffin, Shadowhawk, Dragon, Catapult, Cataphract, Victor, Stalker and Atlas (2/4/3/3). I am actually fond of the idea of a total drop weight limit but that may be from playing a lot of MechCommander. What would be awesome is if we were matched, put into a lobby with integrated comms so we can talk to our PuG team to choose and coordinate our mech choices and customize them to the mission goals (if we had more mission-like matches with various goals). Of course this might be from playing a lot of MechWarrior and MechCommander in the past

[Cimarb]

TheCaptainJZ, on 28 May 2014 - 09:12 AM, said:

Yeah, it's true we need something more than ELO but what else to use is the question. I personally don't like the strict 3/3/3/3 rule only because it makes the enemy team composition predictable and less realistic and for me that's less fun. Of course, a team of all lights against a team of all assaults is not going to be fun either. I'd rather have a "loose" 3/3/3/3 rule where the MM can pick a mech on the border of classes or a neighboring class.

In the VLOG that was just released, Russ mentioned that they are already trying out 4x4 (so up to four of any class), so I think they will have a flexible version pretty soon. I'm sure Karl will have a lot more to say on that point, if he is able.

[Goose]

The point of quad-threes (or whatever we wind up with) is to both prevent Wolf Packs in Conquest, and Th'ah Heavy Jumbo Offensive Line in the other modes …

… As much as I dislike Skirmish, it would seem we have a little too much control over the scenario in public matches …