1911 replies to this topic

[Karl Berg]

Gyrok, on 12 August 2014 - 05:37 PM, said:

Just for the sake of visibility, because I would really like to know...

Gyrok, on 12 August 2014 - 07:40 AM, said:

Karl, I have a question about the IS versus the Clans data...if you could oblige me that would be great.

1.) What was the ELO predicted win % for the Clans in those matches?

2.) What is the acceptable variance of a standard match for prediction versus outcome in your ELO algorithm?

I would like to know this, because if ELO predicted a 90% win ratio for the Clans based on ELO alone, then all is right in the world, and clan mechs are well balanced.

Found it!

1.) I would have to dig back through the old telemetry to produce the actual value. I was involved in facilitating the tests for design, but I wasn't really much involved in the evaluation of the results; otherwise I'd likely be able to say more here. I do recall from watching the live data that the average Elo deltas in solo queue spiked up from roughly 40 to closer to around 100. This is a natural outcome based on the fact that the matchmaker was much more constrained in how it could organize players to produce fair matches.

2.) The answer to this lies in Elo's algorithm itself. The system iteratively converges on skill values based on his probability curve. If you exceed or fail to achieve the expected win percentages, the algorithm forcibly moves your skill value to compensate. The end result is, if two teams were to play each other an infinite number of times, and team A was predicted to win 60% of the time, team A would actually win exactly 60% of games.

I think design has pretty decisively concluded that there is a need to re-balance clan and inner-sphere mechs at this point however. They set out with some fairly aggressive restrictions for clan tech at the outset, hoping this would be sufficient to balance the two factions. I was intrigued by pursuing 10 v 12 solutions myself for a time, especially since it's lore-based. I've heard some pretty compelling reasons why this is not a good approach since. Probably the strongest argument was that a 10 v 12 solution would force us to balance the factions such that a clan mech was roughly 1.2 times as effective as an IS mech. That's quite a bit of power creep, and would mean that all those IS mechs that players have purchased with MC would be devalued as a result.

[Karl Berg]

Heffay, on 12 August 2014 - 06:40 AM, said:

Dear Mr Berg,

Recently there was a post from ...Niko(?) where they mentioned they were spinning up a few more servers to deal with capacity issues that were manifesting as more rubber-banding or latency issues. Something like that.

How much spare capacity do you have? Do you spin up a new blade & ESX host and start off another instance? Is this capacity standing by as needed, or do you Autodeploy game servers?

Just curious as to the underlying architecture and what degree of automation there is around dealing with capacity issues. Thanks!

Love,
Heffay

We have very little automation at this time, as we don't really save much in terms of cost by idling hardware. At the old datacenter, provisioning a new server could take quite some time, and was dependent on a chain of work orders managed by externals. Now we can provision new hardware within 24 hours, but that provisioning is a minimum one-month term as far as I understand. We don't run one instance per blade however. Each blade can run about a maximum of 40 concurrent dedicated server instances. I would love to be able to autodeploy servers based on population levels though. That would be pretty darn cool. If our approach were more cloud oriented, and the servers more amenable to running on VM's, we could likely implement this without too much grief.

We did run some experiments in the old data center with respect to adding/removing capacity, and it's impact on the rubber banding. Unfortunately simply throwing servers at the problem was not enough to make it go away. It is most definitely a software issue, and we will finally resolve this issue in code.

[Karl Berg]

Jody Von Jedi, on 11 August 2014 - 05:24 AM, said:

Is Karl on vacation?

It's been a while since he posted to this thread.

July 25th was his last post.

Sorry Today was one of the first days I've had off in a couple months.

[Karl Berg]

Stealth Raptor, on 09 August 2014 - 06:49 PM, said:

I have a question about unit names. As i have heard Canon unit names have been reserved by pgi. My unit has represented one of the canon units throughout many previous mechwarrior games, and we want to make sure it isnt taken. my question is with CW phase 1 coming out, are the canon unit names being saved, and if so can we get a list so we can know to grab it if it is available? We really dont want to lose the opportunity to grab our proper name if it will become available

And I'm too late to add much here. Hopefully, between the pre-reservation thread, and the announced canon names list, you got all the answers you needed. I am actually quite excited by units. Underlying everything, they represent the first shared persistent writable data in our system (writable data belonging to multiple users). This forced me to finally finish off some of the sharding tech that's been pending in our architecture for so long.

[MischiefSC]

Karl Berg, on 06 September 2014 - 06:08 PM, said:

Yes, it effectively alternates adding to each team, as long as the size of the groups being added are roughly equivalent. In the case where an 8 man is added to team 1, and a 5 to team 2, then the next team to receive a group would be team 2 since it is still the smaller.

And now this is exactly where it gets complicated.. Yes, the average Elo of the seed group is saved off. This become the baseline skill level around which the entire game is built up on. There are also obviously multiple competing constraints that the matchmaker is attempting to simultaneously satisfy, including group size, Elo, weight class, game modes, region, and factions when we enable that feature. Most of those constraints require some form of tolerance decay, or release valve as I describe them to design. The decay functions are exponentials, with constant start and stop inputs, that vary with respect to time (age). That evaluation of a given decay function is quantized onto an output range, and these are fed into a search algorithm that locates potential matches and returns the most desirable. All of these parameters are tunable on the fly from our matchmaking command center.

This somewhat analog approach to balancing simultaneous constraints seems to work quite well in practice. We can easily make small tweaks to inputs to subtly change our desired biases in match outputs. There are a few constraints that always take priority. These are what we refer to as hard-constraints, as opposed to the softer constraints like Elo, or the number of heavy mechs in a match. Those hard constraints are currently team size, balanced weight class matching (although this switches off after 4 minutes searching in group queue), region, and game mode.



It's a random dice roll, using the c standard random function, seeded with system time, every time a server boots. There should be no magical stride that sees you getting placed on the same server on the same map repeatedly. There have been times where I could swear there was some selection bias in the system myself though. One of the engineers went so far as to write up a quick simulation of distributed RNG's like we're using just to see if there was a loss of entropy when compared to a single random number generator. We found that the distribution using a distributed system was less random. Not terribly surprising I guess. Anyways, all of this has reminded me to go back and change this so that it uses a centralized RNG; so thanks!

Oh dear. Not to start the conspiracy theory train rolling but are you saying that there are actual people broadening or tightening specific criteria on matches individually? Please tell me yes, that would actually be pretty hilariously awesome. Not that I'm saying anyone would intentionally ghost particularly annoying players to feed them loss after loss after loss but the idea that it's hypothetically possible....

Warm fuzzy.

How about the pug queue? Does it start with a pair and build out? While I get that it's got to decay out criteria restrictions does it have a priority? Also does it factor Elo as a 'preference is for people in the same range' or 'grab high/low to target from oldest to youngest queue time' out of the gate?

I get that by necessity you're going to have mixed ranges of Elo in each match. At the risk of drawing flame I'm even in favor of that to a degree; otherwise it would be terribly difficult to move your Elo out of lower levels. Is there a preference for keeping people in a range though or is it just 'grab 2 seeds, grab 4x3 matches with approximate Elo balance to match target' out of the gate?

[Cimarb]

Karl Berg, on 06 September 2014 - 10:59 AM, said:

That's an interesting suggestion. I've been looking very closely at the ranges and standard deviations of Elo within matches myself, with a definite goal towards reducing standard deviation as much as I possibly can without adversely affecting the other matching criteria. The good news is I do have a set of pending changes that appear to help a great deal when tested against current production data. I'm hoping to get these tested and released as soon as I can.

Sorry guys, I need to step out for a bit.

That, and your answer to my question, are both GREATLY appreciated.

Thank you for all the answers, especially on your first day off in so long.

[Modo44]

Any ETA on matchmaker brackets/buckets/leagues/whatever? Notice how I am not asking "if".

[Sarlic]

Karl, you're great. This is what i would like to see from more developers. This is why i come to the forums.

Communication is all i want. Clear, crisp and honest communication.

Edited by Sarlic, 07 September 2014 - 04:49 AM.

[DeathlyEyes]

Heffay, on 05 September 2014 - 04:26 PM, said:

One game out of 10 billion, this maybe happens.

Every single one of my matches is like this......

Edited by DeathlyEyes, 07 September 2014 - 06:21 AM.

[TKSax]

Karl,
Thanks for answer my other question now one about MWO.

I asked this questions to a lot of people and no one was sure, and it seem like a simple questions.

Does the % of mech's in each class include both ques or just the solo que?

Thanks

[Karl Berg]

MischiefSC, on 06 September 2014 - 07:49 PM, said:

Oh dear. Not to start the conspiracy theory train rolling but are you saying that there are actual people broadening or tightening specific criteria on matches individually? Please tell me yes, that would actually be pretty hilariously awesome. Not that I'm saying anyone would intentionally ghost particularly annoying players to feed them loss after loss after loss but the idea that it's hypothetically possible....

Warm fuzzy.

How about the pug queue? Does it start with a pair and build out? While I get that it's got to decay out criteria restrictions does it have a priority? Also does it factor Elo as a 'preference is for people in the same range' or 'grab high/low to target from oldest to youngest queue time' out of the gate?

I get that by necessity you're going to have mixed ranges of Elo in each match. At the risk of drawing flame I'm even in favor of that to a degree; otherwise it would be terribly difficult to move your Elo out of lower levels. Is there a preference for keeping people in a range though or is it just 'grab 2 seeds, grab 4x3 matches with approximate Elo balance to match target' out of the gate?

Modo44, on 06 September 2014 - 11:15 PM, said:

Any ETA on matchmaker brackets/buckets/leagues/whatever? Notice how I am not asking "if".

Not purposefully by any means. Some players are just much harder to match than others, so they end up sitting in queue for much longer and produce games with different properties than most others.

Solo queue starts with the single oldest request that is hard-constrained from ever matching any other game the matchmaker is currently working on. It's hard to explain without the use of some graphics, but essentially that original requests Elo determines the range of Elo's that can be examined as the game ages. The difference between the two pending teams Elo's determines where within that range the matchmaker will start looking for it's next match.

I would like to see people stay within suitable ranges where appropriate. Accomplishing that goal has it's own complexities though. It's not as simple as saying 'Elos 2000+ can never match with anyone lower'. We tried something like that, briefly, and found it lead to terrible queue starvation for those high Elo players. Keep in mind we've split our queues quite a few ways for players already. We have solo and group queue, we have hard-constraints for game modes, we have strict weight class caps (no more than 4 of any weight class in solo queue), as well as some others that currently are not as big a factor. That set of 2000+ Elo players is actually very small, limited to a handful of the best players from the best comp teams. Currently that's not enough to reliably kick off games within themselves 24/7 in all queues.

[Karl Berg]

DeathlyEyes, on 07 September 2014 - 06:21 AM, said:

Every single one of my matches is like this......

When the matchmaker is forced into large ranges of Elo's, by my observation, almost always both teams have high standard deviation, as opposed to a single team having high standard deviation facing off against a low standard deviation team. That doesn't excuse the poor matches; but that is the current behaviour of the system.

TKSax, on 07 September 2014 - 08:50 AM, said:

Karl,
Thanks for answer my other question now one about MWO.

I asked this questions to a lot of people and no one was sure, and it seem like a simple questions.

Does the % of mech's in each class include both ques or just the solo que?

Thanks

Both queues.. When I first wrote that, the two queues actually overlapped for 2, 3, and 4 mans. I completely separated the queues a few hours after launch of the new matchmaker at Russ's request. So yes, both solos and groups are thrown together when producing those queue percentages that are shown in the front end.

[MischiefSC]

Seriously Karl, can't say enough how much we appreciate your time here. I debate the mm topic often and this sort of stuff is invaluable to know.

So on Elo variance and the like how much benefit would it be to remove game mode selection?

Is the primary issue the 2k+ folks?

What it game mode was more a preference than a hard restriction? The problem I see with high/low to target involving 300+ variance on Elo on a team is it starts to make Elo progression exponentially more difficult the further up the curve you go. Part of the perk of an Elo progression is a gradual curve on improvement at an individual level even if your actual position on the curve is dramatically steep. If movement out of majority ranking involves heavy carry every match you're going to get stuck in the middle.

This gets more problematic when you account for us not knowing if this match is a gain or loss opportunity for us. I agree we don't need to see our Elo scores but can we see the projection for the match?

Also opting in and out of Elo matches. If I'm getting basics in a nova I'm not performing at expectation. Mm performance is going to skew match results.

Opt out out Elo, or opt in. Game mode a preference not a limit. Wouldn't that give better overall match results and more accurate Elo?

[Sarlic]

Karl Berg, on 06 September 2014 - 07:01 PM, said:

Sorry Today was one of the first days I've had off in a couple months.

Question: did you enjoy your day off? Sometimes you get some fresh (atleast i get it) idea's on your day off.

[Modo44]

Karl Berg, on 07 September 2014 - 11:42 AM, said:

That set of 2000+ Elo players is actually very small, limited to a handful of the best players from the best comp teams.

You want a number of brackets with the same number of people in each, not static Elo ranges. Trying to satisfy the tiny outliner groups will probably never be possible, but you can make matches a lot more sensible for almost everyone. Make the bracket borders variable, and adjust periodically to account for Elo moves.

[Steinar Bergstol]

Seriously, Karl, I think this thread and your efforts here have done more good for PGI's PR and standing among the playerbase than anything else since closed Beta. Thank you for being a virtual patron saint of patience and putting up with us and our questions. My hat would be off to you, sir, if I had a hat.

[Mawai]

Chronojam, on 06 September 2014 - 12:02 PM, said:

Hi Karl; it's great to see you're still answering fan concerns here. I tried asking Bryan Ekman a few questions directly the other night in that thread since they were a bit related to both MWO and PGI's other projects, but as of this morning,


So focusing on MechWarrior Online, I'd like to check into some of the statements we've been given, just to help set the record straight or at least clear up some discrepancies.

In 12/2013 we were told by Russ that MWO had a staff of 45 dedicated developers; we'll use this as our baseline.

In 5/2014 fans were concerned over slow progress, caused by understaffing forcing the need to shift e.g. the entire environment team to work on clan mech design.

Later in 5/2014, Russ bragged that PGI had grown 25% (from 11/2013 to 5/2014) so MWO fans should not worry about continued staffing problems.

In 9/2014 Bryan explained that MWO currently has reduced a staff of 35-40, which now includes external contractors.

Hopefully, as these are public statements, you won't have any trouble filling in the gaps for us. Despite bragging of 25% growth over six months, four months later MWO's team appears to have shrunk by 10-20% compared to baseline -- in conjunction with a reliance on external contracted labor instead of dedicated in-house developers.

Were these statements true? Is a reduced MWO workforce, dependent on outside labor, now struggling to keep up with old development timelines?

To be honest ... I am not sure that MWO has ever had more than 6 developers or so working on the project code so I have no idea how they can possibly support development of MWO in conjunction with a new project without additional resources. I am seriously dubious about any claims they will make at the moment since their past track record is ridiculously bad. (The classic CW within 90 days of open beta comes to mind).

Here is a post from Karl outlining the status of engineering resources on 8 may 2014 ... at that time there were only 8 engineers working on the game ... "That said there are 3 systems engineers, 3 gameplay, and 2 UI."

http://mwomercs.com/...ost__p__3360690

How can 8 people ... only 3 of whom are gameplay engineers ... actively develop two major titles? MWO development is already too slow in the view of many players ... how can it do anything but slow further if engineering resources are diverted onto a new project?

[Gyrok]

Karl Berg, on 06 September 2014 - 05:31 PM, said:

Would you mind reposting your question for me Gyrok? I'm going to continue working my way back through the question backlog, but a repost would help me answer your question sooner. Thanks!

How often were the Clans the higher ELO? What impact would that have on the results of the matching?

If you can give me even rough estimates off the top of your head that would be great.

Also, another point about 10 vs. 12.

You do not have to make clans 1.2 only 1.16-1.17.

This is predominantly because you would be balancing 5 vs 6. I think a balance approach that revolves around focusing on the possible total output of the team of 12 versus the team of 10 is a more prudent approach versus trying to balance on a 1 to 1 level.

When I did some balance work in the past on powerset systems, I, personally, found it easier to balance something as a sum of the parts in a case like this, versus trying to balance something on an individual case by case basis.

The one drawback there is that you do have to take into consideration that minor adjustments will have a bigger impact on the overall performance of the team in general. This often presents itself in ways that would not be immediately evident when making adjustments, and does require LOTS of playtesting to get right. Basically, we used to run a LOT of test server time for players to be able to test out new builds and provide feedback. I personally feel that this type of development where a build with some adjustments can be hot fixed to the test server to gauge the results of the adjustments would be a more beneficial model for MWO as a whole.

10 vs 12 can be done, I think people may be looking at it from the wrong angles though. If you are curious, I would be more than happy to bounce some ideas your way and you guys can use any of them if it contributes to a 10 vs 12 environment that the playerbase as a whole is happy with and your development team sees no issues with.

Let me know on that end if you want any of those ideas. Because I would be willing to help out in any way I could on a volunteer basis...

Edited by Gyrok, 08 September 2014 - 11:42 AM.

[Lordred]

@Karl

You teased us a while back with the mention of a 4k res pack, any update on that by chance?

[Lordred]

Just imagine this, with more detail.