1911 replies to this topic

[Rebas Kradd]

Updated 2:45p PST 5/4/14

Since this thread has rapidly gone in different directions and become a smorgasbord of developer insight (thank you, Karl and Brian!), I'm going to try and organize this thread a bit by topic, Helmer-style. Let me know if I screwed up any links.


***Also of interest: Karl on Reddit***


Development

Response to Major Community Concerns

Karl's Work Responsibilities, Overall Development Status

Links to Karl's presentation materials

Impact of UI2.0 on development.....on Community Warfare

Glimpse into PGI development pipeline

More on pipeline, Karl's physics experience (NEW!)



Programming Talk

Impact of autogenerated code

PGI coding practices, lessons learned

Learning curve for autogen code

Software regression (AKA "ghosts in the machine")

Internal testing procedures and tools.....custom unit test tool en route for devs

Maintaining of code branches (NEW!)



Server Talk

Server OS and developer ages.....CryEngine's role

More tidbits on dedicated servers

Challenges of Public Test Server

Matches run on bare-metal servers, not virtualized servers, due to latency and cost concerns (thus the charging for private matches).....theorizing on business decision of monetizing Launch Module

Lots of info on Host State Rewind (HSR)

Explosion queue fix should affect weapons besides SRMs (NEW!)

Addressing effect of server tick rate (NEW!)

On role of ping and speed cap (NEW!)



TrackIR

TrackIR Progress.....update.....process talk.....breakthrough, handed off to Gameplay team

Challenges of Oculus Rift Integration

Oculus Rift and cockpit glass



ELO and Matchmaking

On reluctance to split queues, reveal ELO.....even on teamwise basis

ELO usage still fluid

Groups sensitive to ELO changes.....common group ELO telemetry.....accuracy of outcome prediction

Willingness to explore group dynamics.....more.....philosophy for making changes

Effects of time zones and seasons

Pairwise tracking of ELOs

ELO currently separated by weight class.....but difficult to matchmake by weight class.....or to analyze needed weight classes in a drop.....or to calculate team deltas (NEW!)

On the probabilities of a 26-game losing streak

ELO buckets strict, more on matching procedures (NEW!)



Out-of-game Documentation

On PGI-maintained wikis (NEW!)

Will look for devs to act as fact-checkers (NEW!)

Such information time-intensive and constantly changing (NEW!)

On what kind of info to wiki, player retention tracked by PGI (NEW!)



Upcoming Stuff

Data mining tools in pipeline, VPN tweaking being reviewed

Brian Buckton on future social fixes.....more...

Role of already-released features in Community Warfare

VOIP work underway.....intended as user tool, not gameplay mechanic

PGI aware of color-blindness issue, 3/3/3/3 toggle coming with Launch Module.....color-blindness work on hold

Concerns with FOV slider (now confirmed by ThePlan)

Status of MASC



Miscellaneous

On draw distances and disappearing objects.....being reviewed....."weirdness" confirmed by rendering team.....theory on terrain hitboxes

Potential tradeoffs with changing nature of autocannons.....damage changes require careful testing

Hidden critical hit system in place

Positive experiences with NGNG.....Friday session was off-the-cuff

Company notification practices

Meaning of "upside-down guy" in-game icon is a mystery

Physics of falling objects in MWO (NEW!)

Niko Snow: Will try to keep ThePlan updated regularly (NEW!)

Direct weapon/chassis balancing preferred to telemetry-driven balancing (NEW!)

Dennis de Koning on mapmaking (NEW!).....on mech creation (NEW!)



ORIGINAL POST:
-----------------------------------------------------------------------------------------------------------------------------------

So I spotted this amongst Bryan Ekman's recent Twitter blurbs...

http://schedule.gdco...ssion-id/825586

It's PGI's own Karl Berg getting his own session at a game developer's conference in San Francisco this upcoming March. Congratulations Karl, and we miss your posts around here.

Apparently he's going to be discussing something called asynchronous server architecture, which is described here in a very interesting way...

Quote

Stateless and asynchronous software designs are extremely scalable and efficient, and this efficiency can be crucial in controlling server costs with online free-to-play titles. Unfortunately, asynchronous software also introduces significant complexities, which decreases developer productivity, and significantly increases the risk of bugs slipping into live production environments. This talk describes some of the major challenges associated with these architectures, as well as specific techniques used in MechWarrior Online to compensate for these challenges.

I think it's fair to say that we've all been wondering about the reasons behind PGI's development slowdowns and missed deadlines in 2013. We've gotten a few nuggets involving manpower shortages and the UI being tied directly into the game and forcing them to patch the entire game just to have a sale. But I'm not sure it's given us a clear picture yet.

Now I'm getting kind of curious about how responsible this asynchronous server architecture stuff (which sounds like a relatively new thing in client design) has been in the constant slowdowns. It makes it sound like PGI is kind of pioneering in this area and learning as they go.

Could we perhaps get Karl to put together a post for the community describing the role of this stuff? I realize it isn't easy to break it down for us laymen, but Karl has shown talent at doing just that. I would look forward to such a post.

Edited by Rebas Kradd, 04 May 2014 - 12:46 PM.

[Karl Berg]

Hi Rebas,

I'd love to talk about it; somewhat obviously since I'm presenting on this topic at GDC . I have agreed to not share any talk details prior to the conference itself as a condition of speaking, so I can't say too much about the topic of the lecture beyond the synopsis I've already had posted to the GDC conference scheduler.

As to the responsibilities of the systems I'm speaking on, well if everything is working perfectly they should be completely invisible. They manage everything from storing, retrieving, and updating your profile and inventory data, to session management, routing chat and private messages when you're in the front end, matchmaking, high-level dedicated server management and load balancing, billing, accounting and audit trail work, telemetry gathering and even generating those heatmaps you sometimes see posted around the site. Our system is custom built for MWO and has been in continuous operation since the launch of friends and family way back. In all that time we've only had 2 server crashes, one due to a threaded race condition which was hotfixed immediately after it occurred, and one in the OpenSSL library which we use during session initiation.

I won't be talking about production or scheduling issues; production and project management would be in a much better position to respond on these issues. I can say that our developmental server code is a good 6 to 8 months ahead of production at this time, mostly due to the enormous bubble of pent-up features backlogged by the UI team trying to get UI 2 out the door. Hopefully with UI 2 finally getting released in a couple weeks, they can start slapping UI together for all the new stuff and we'll get that feature backlog finally cleared out at long last.

[Charles Fornton]

Lovely and mature thread, Shame it was moved here.

[Heffay]

Charles Fornton, on 26 January 2014 - 11:15 PM, said:

Lovely and mature thread, Shame it was moved here.

It's a fine place for it. And it's very interesting stuff to read. I'm going to try to watch his presentation as geeks talking geek talk makes me all kind of happy in the pants.

[Rebas Kradd]

Heffay, on 27 January 2014 - 05:01 AM, said:

I'm going to try to watch his presentation as geeks talking geek talk makes me all kind of happy in the pants.

Will it be webcast? I doubt I'd be able to understand half of it, but you never know...

[Tekadept]

Karl Berg, on 26 January 2014 - 07:31 PM, said:

Hi Rebas,
I have agreed to not share any talk details prior to the conference itself as a condition of speaking, so I can't say too much about the topic of the lecture beyond the synopsis I've already had posted to the GDC conference scheduler.

Sooo GDC talk is over, time to spill

[Heffay]

Tekadept, on 25 March 2014 - 07:59 PM, said:

Sooo GDC talk is over, time to spill

Yup! Get in here, Karl!

[Karl Berg]

Hi! Yes, all done. The slides can be found here:

http://www.gdcvault....in-Asynchronous

It doesn't include all the nifty animation, which I spent quite a bit of time on. Unfortunately, the link to the example code was emailed to me by Fox and copied out of outlook when I was in San-Fran, so directly clicking the example code link tries to open my outlook in order to load the URL. Nice.

So here is a direct link to the example code instead:

http://mwomercs.com/...arl/GDC2014.zip

That includes visual studio 2010, XCode 5, and CodeLite solution / project / workspace, and builds on all platforms as far as I am aware.

[Karl Berg]

Sorry all; someone informed me that this thread had arisen from the grave, but I had a bit of a queue this morning, and had to post and run the above.

If any of you have questions about the presentation, I'd be more than happy to discuss. I'll also field any questions about MWO architecture, with the caveat that I won't be able to share anything that ventures into NDA / trade-secret territory.

[Heffay]

Karl Berg, on 27 March 2014 - 10:52 AM, said:

Sorry all; someone informed me that this thread had arisen from the grave, but I had a bit of a queue this morning, and had to post and run the above.

If any of you have questions about the presentation, I'd be more than happy to discuss. I'll also field any questions about MWO architecture, with the caveat that I won't be able to share anything that ventures into NDA / trade-secret territory.

Are your back-end servers Linux then?

... You just caused my brain to effectively shut down. You say to ask you anything, and my brain just went into a race condition with all the questions I want to ask!

Who is the oldest person working at PGI, and how old are they?

...

SEE? I GOT NOTHING!

[Karl Berg]

Yes, our backend servers are linux. Well, and windows too, for the dedicated servers that run the game instances.

Oldest person.. hrm.. I'm not actually sure! I don't even know how I'd reasonably go about asking everyone their age to answer that, considering how some people get upset over questions like this on occasion.

I'd say the studio's average age is late 20's to early 30's, hopefully that's both somewhat accurate and good enough!

[Rebas Kradd]

Karl Berg, on 27 March 2014 - 10:52 AM, said:

Sorry all; someone informed me that this thread had arisen from the grave, but I had a bit of a queue this morning, and had to post and run the above.

If any of you have questions about the presentation, I'd be more than happy to discuss. I'll also field any questions about MWO architecture, with the caveat that I won't be able to share anything that ventures into NDA / trade-secret territory.

I suppose I'm just really searching for understanding on what happened to 2013. That story is pretty rough around the edges for us, other than the "It was all UI2.0" disclaimer (which, I might add, was sorely undercut when UI2.0 debuted incomplete). A lot of folks are frustrated as to the relative lack of progress last year on the game and love to strut around talking about how there are loads of other F2P games out there (Hawken and War Thunder, for example) that get twice as much done with half as many programmers or what have you. I personally am a supporter of PGI and would like to see such comments put in context.

So when I read the synopsis of your presentation and noticed your mention of "decreased developer productivity" in regards to stateless server architecture, I thought it might shed some light on the game's development story.

You also mentioned that PGI is mostly C++ programmers, and I glean this introduced some challenges with the methods you were trying to use.

Is it accurate to say that MWO has been pioneering some brand-new(?) software methodologies (asynchronous architecture, autogenerated code) that you've had to learn on the fly? This is the sort of thing that would really put the game's story on firmer ground with the community, although I realize that you can't break any NDAs...

Edited by Rebas Kradd, 27 March 2014 - 01:40 PM.

[CyclonerM]

Wow.. How much time did you spent to create all the animated slides?

This is the kind of things that make me want again to try learning to make games.. But i love too much to play them

[Sandpit]

Charles Fornton, on 26 January 2014 - 11:15 PM, said:

Lovely and mature thread, Shame it was moved here.

Off Topic isn't a graveyard, it's not a bad place for anything

It's definitely some interesting information. Going to take a bit to read through it all. Thanks for all the info Karl!

[Karl Berg]

@Rebas Kradd

Hrm, well 2013 was indeed a weird year. UI 2.0 has been in the works for that entire period, which was pretty much blocking any major new feature releases. It's not really my place to discuss exactly why it took as long as it did. That's best left to a post-mortem done by production or the UI programmers. My talk was entirely related to the Systems team, and MWO server tech; and has little to nothing to do with UI 2.0, beyond the fact that UI 2.0 is now starting to use our systems to autogenerate small amounts of action script and C++ binding code to simplify their lives.

The effect on the systems team was we could implement all the new features we wanted server-side, but without a UI to expose them they're not of much use. So instead we built up new functionality, like the recently released achievement systems, in isolation in a branch, and then shelved the work to move on to the next thing, waiting for the time when the UI team could come along and write the presentation layer for public release. We did a very large amount of work in this manner, in fact the April 1st patch will launch a pile of server-side scalability work we did to support CW (still no presentation layer yet, sorry).

@CyclonerM

Far too much time sir, far too much... They were neat animations though; I'm highly impressed with Keynote.

@Sandpit

You're very welcome; hope you enjoy it!

[Alaskan Nobody]

CyclonerM, on 27 March 2014 - 01:21 PM, said:

This is the kind of things that make me want again to try learning to make games.. But i love too much to play them

I feel ya with that one - working on learning some programing so I can do it anyways, as I don't have a lot of actual workable skills.

Sandpit, on 27 March 2014 - 01:52 PM, said:

It's definitely some interesting information. Going to take a bit to read through it all. Thanks for all the info Karl!

When you are done could you give me a hand?
Partially because breakup (as in the seasonal change between winter and construction) has me feeling less than brilliant - and I have no idea what most of it meant.

[Tekadept]

Karl Berg, on 27 March 2014 - 12:18 PM, said:

Yes, our backend servers are linux. Well, and windows too, for the dedicated servers that run the game instances.

Oldest person.. hrm.. I'm not actually sure! I don't even know how I'd reasonably go about asking everyone their age to answer that, considering how some people get upset over questions like this on occasion.

I'd say the studio's average age is late 20's to early 30's, hopefully that's both somewhat accurate and good enough!

It sounds like rather then one dedicated server" per match, you have a dedicated server run multiple instances of matches, what is the ratio, ie how many matches per dedicated server, how many dedicated servers can a "physical" server run. or do you just have it so that one "Physical" server runs one dedicated server, and al the match instances under that?

I Know it has been mentioned but everybody has given up on ever having regional servers, but "if" there was to be, is the current server architecture/ Matchamaker written to allow that , or would it require modification

[Heffay]

Oh, I have a legitimate question, thanks to some of Tekadepts points. Do you use any virtualization? At work we provide servers on demand for customers, and can spin up VMs (Windows and Linux both) in very short order to deal with capacity issues. Do you use virtualization (i.e. vmware or the like) in your environment, or is it all physical servers?

[Karl Berg]

Yup, we run several virtual servers.

Yup, we run multiple dedicated server instances on a single physical server. The ratio is about 24 instances per blade, but we monitor blade utilization and adjust this as needed.

And yes, our matchmaker and server architecture will have basic support for regionalized dedicated servers starting with the April 1st patch. For future requirements lets say. As for actual regional deployments that would make use of this newly added support, I can't say anything at this time.

[Mycrus]

Nice Karl this is the kind of communication that russ Bryan Paul tribe should learn from... say what can be said and have the users digest it according to their capacity.