112 replies to this topic

[Rebas Kradd]

Why Making Multiplayer Games is Hard: Lag Compensating Weapons in MechWarrior Online

http://www.gamasutra...rior_Online.php

Quote

Since the client is predicting everything, can’t the client just tell the server they hit the target or not? The simple answer is yes, that is a possible solution; however, there is a very big caveat. The moment the game client is allowed to tell the server anything the potential for some form of cheating is created. On PC, a game client can trivially be modified and a smart cheater will modify the game client to send hit messages whenever they want, even when they aren’t near or can’t see targets. As such, this notion of client authority, especially in the case of dealing damage, is an incredibly risky proposition.

Which is why MWO goes with server-side authoritative approach.

Quote

In the case of MWO, it is not sufficient to only store past positions and orientations. Each component of the Mech can be damaged separately, and this is a very important gameplay mechanic, so the animation state is also relevant. This must be remembered and rewound on the server as well.

Quote

Projectile weapons work very differently from trace fire weapons. They fire a piece of geometry, like a slug or missile, into the world and this geometry sails through the air and deals damage once it impacts a target. A simple rewind will not work in this case. If we used the same approach as trace fire weapons and assumed the projectiles travelled at an infinite speed, the server might incorrectly detect the projectile struck a target immediately, even though the target could have potentially moved out of the way before it hit them. And even worse, players intentionally lead shots ahead of targets for projectile weapons since they are using their judgment to predict where the projectile will be by the time it reaches their target. In this case, a player with perfect aim, and a server with perfect rewinding will cause the player to always miss. We need a different approach.

Quote

What we would like to do is fast forward the projectile on the server to catch up to the position where the client has predicted it will be. If we can make the prediction of the client correct, then we will have compensated for the lag on the game client.

What I'd be interested in, is hearing more specifics about how this sets MWO apart from other games - how much more calculations are being run on the server than your average FPS, what demands it places on map design, and examples of how it affects other systems. I know these questions are a little broad...

Edited by Rebas Kradd, 07 September 2016 - 12:34 PM.

[Revis Volek]

Rebas Kradd, on 07 September 2016 - 07:37 AM, said:

Why Making Multiplayer Games is Hard: Lag Compensating Weapons in MechWarrior Online

http://www.gamasutra...rior_Online.php

[/size][/font]

Which is why MWO goes with server-side authoritative approach.







What I'd be interested in, is hearing more specifics about how this sets MWO apart from other games - how much more calculations are being run on the server than your average FPS, what demands it places on map design, and examples of how it affects other systems.

We cant even get a player count, i really doubt they will or even can divulge this info to you in a concise manner.


But maybe im wrong.

[Iron Heel]

Edited:
Off topic and irrelevant rant.

Edited by Iron Heel, 08 September 2016 - 04:35 AM.

[Neema Teymory]

Hey Rebas,

Rebas Kradd, on 07 September 2016 - 07:37 AM, said:

What I'd be interested in, is hearing more specifics about how this sets MWO apart from other games

I can't speak on behalf of all other multiplayer games, but in my experience, predicting and lag compensating projectile weapons is generally a much harder problem to solve and so is usually avoided. This would be something that sets MWO apart. As I mentioned in the Gamasutra article, the nature of MWO movement gives us the opportunity to do something smart to lag compensate projectile weapons efficiently. In a twitch style FPS where you can instantly change the direction of your velocity in a single frame, the non-iterative solution I have described will not work correctly, and you will likely need to fallback to some kind of iterative solution which can be very complicated and/or expensive.

Rebas Kradd, on 07 September 2016 - 07:37 AM, said:

how much more calculations are being run on the server than your average FPS

Again, it's tough for me to say what all other games do, but I can give you a sense of the extra work MWO does in order to lag compensate weapons. When you fire, the server needs to rewind all 'eligible' targets--including friendly targets. This is because if you don't rewind all targets, you may strike a 'present' target and incorrectly detect a hit on the wrong target. Rewinding 23 mechs (because you don't rewind the shooter) every shot is really inefficient, so what we do to optimize this is build a set of eligible candidates based on a bounding box that encapsulates the entire rewind period for a target. You can imagine a rectangular box that stretches along the velocity vector of the target. It extends from the position of the rewound target position ping seconds ago to the present target's position. We can ignore all targets whose stretched bounding box does not intersect the firing vector. This is a very important optimization, because the actual rewind operation itself is relatively expensive, and this optimization often culls the number of targets that need to be rewound to one or two. The worst case is when every mech is lined up in front of you when you shoot. All these targets would need to be rewound.

A projectile rewind is this same rewind check plus a little more work. Instead of rewinding by ping, we need to rewind by an adjusted ping that is based on a line test of an adjusted velocity vector against each potential target's geometry. This is better explained in the Gamasutra article, so I won't repeat everything here. This amounts to 23 additional line checks against target geometry. We can't only use the bounding box of the target here, because we need an accurate hit time to get the correct adjusted ping value. Of course, the bounding box is used as a broad phase optimization.

So to summarize the extra work the server does for lag compensation:
- For a trace fire weapon: 23 line box tests + N (usually 1 or 2) target rewinds + 1 world ray trace
- For a projectile weapon: 23 line target tests + everything else a trace fire weapon does

Keep in mind this is per shot, and in the case of a laser, this is sustained the entire time it is firing. With all our optimizations, the servers ends up being able to manage this extra work without too much trouble, even in an intense firefight with many people firing simultaneously.

Rebas Kradd, on 07 September 2016 - 07:37 AM, said:

what demands it places on map design

I wouldn't say this has any direct affect on map design. It's still important that maps be within their appropriate budgets to ensure client performance isn't negatively affected. And it is generally ideal if a map isn't designed to clump everyone together into tight corridors. As I mentioned earlier this can make rewind checks more expensive. But generally speaking this isn't a problem that requires adjusting map design.

Rebas Kradd, on 07 September 2016 - 07:37 AM, said:

and examples of how it affects other systems.

This biggest system affected is of course the weapons system. As you can imagine, this system is very sensitive to weapon mechanics. What might seem like a simple change might have serious impacts to the rewinding implementation. If, for example, design wanted to add a warm up animation to a weapon before it fires, this could seriously mess up the timing of the current implementation and would probably need to be handled specially.

The next biggest related system is movement. If design decided to suddenly add a game mechanic that allowed a mech to stop on a dime and turn in the opposite directly, this would have drastic consequences for the rewind implementation--even to the point of breaking it.

The physics system is also affected. Depending on the way your physics engine works, it may not like the idea of moving a single object outside the regular physics simulation for a rewind. This was the case for our version of CryENGINE and we had to make some adjustments to get this working correctly.

I hope that answers your questions

[Quicksilver Aberration]

Neema Teymory, on 07 September 2016 - 10:21 AM, said:

I can't speak on behalf of all other multiplayer games, but in my experience, predicting and lag compensating projectile weapons is generally a much harder problem to solve and so is usually avoided. This would be something that sets MWO apart. As I mentioned in the Gamasutra article, the nature of MWO movement gives us the opportunity to do something smart to lag compensate projectile weapons efficiently.

Not to derail this too far, but I do have a point of contention here in that games that don't use all hitscan weapons are doing this, as overwatch has a similar system in place to deal with lagshots, the only difference is that compared to this game, theirs feels a lot nicer to play:

Now it may not necessarily be the netcode that makes theirs better but may be related to collision detection/resolution that makes theirs feel better when playing but either way, they don't have as many complaints about netcode.

Edited by Quicksilver Kalasa, 07 September 2016 - 10:39 AM.

[Novakaine]

Blue letters!
Someone actual exist there beside the Russ!
See I told ya.

[Bud Crue]

Rebas Kradd, on 07 September 2016 - 07:37 AM, said:

Why Making Multiplayer Games is Hard: Lag Compensating Weapons in MechWarrior Online

http://www.gamasutra...rior_Online.php

Cool read. Not only from a "how does this stuff work" perspective, but a "wow folks at PGI really are working on stuff behind the scenes" perspective.

Rebas, Thanks for posting this.
Neema, Thanks for writing the article.

[Neema Teymory]

Quicksilver Kalasa, on 07 September 2016 - 10:24 AM, said:

Not to derail this too far, but I do have a point of contention here in that games that don't use all hitscan weapons are doing this <snip>

Hey Quicksilver,

Yes, it's certainly true that other games have solved this problem. And as I mentioned, I certainly can't speak on behalf of other games. We make no claim that this is a problem solved only by MWO. In an effort to answer Rebas' question, I pointed out that this might be something interesting that sets MWO apart. But, maybe it doesn't! The main goal of this article was to share some of the challenges that arise in multiplayer development, and to highlight that with a specific example.

[awdwikisi]

Thank you for explaining the rewind mechanic, must have missed it if was explained in CB. This gives insight into why delayed convergence was removed in favor of a smoother 12 v. 12 environment. Though I still miss delayed convergence significantly.

[Dread Render]

As a programmer myself just thinking about the complexity of detecting a hit in a multi user environment where everyone has a different connection speed gives me a headache.
... and whenever I am spectating on another mech I cant help but think about it.
All those projectile trajectories must be tracked and displayed.

I think you guys are doing a great job.


edit---
sorry cant help it.
Now Fix The Damn MM

Edited by Dread Render, 07 September 2016 - 10:50 AM.

[Quicksilver Aberration]

Neema Teymory, on 07 September 2016 - 10:40 AM, said:

Yes, it's certainly true that other games have solved this problem. And as I mentioned, I certainly can't speak on behalf of other games. We make no claim that this is a problem solved only by MWO. In an effort to answer Rebas' question, I pointed out that this might be something interesting that sets MWO apart. But, maybe it doesn't!

That's fair, just wanted to point out that MWO isn't as unique as it used to be .

Having done a short period into studying collision detection I know about the complexity that time/lag (especially the discrete vs continuous problem) introduce when doing these sort of calculations, specifically those involving non-hitscan type weapons.

Edited by Quicksilver Kalasa, 07 September 2016 - 10:55 AM.

[Iron Heel]

I didn't realize MWO and it's development staff have faces down challenges heretofore insurmountable by previous development teams.

No
I'm not in a particularly good mood today.

What are you looking at?
Get back to work.

[Rebas Kradd]

Thanks for the explanation, Neema. Probably means that jump jets require a pretty careful design review, too.

Is pure server-side authority a unique choice amongst CryEngine games, or do other games do this?

[Zordicron]

Neema Teymory, on 07 September 2016 - 10:40 AM, said:

Hey Quicksilver,

Yes, it's certainly true that other games have solved this problem. And as I mentioned, I certainly can't speak on behalf of other games. We make no claim that this is a problem solved only by MWO. In an effort to answer Rebas' question, I pointed out that this might be something interesting that sets MWO apart. But, maybe it doesn't! The main goal of this article was to share some of the challenges that arise in multiplayer development, and to highlight that with a specific example.

Would reducing matches to 8 vs 8 have a positive impact on the efficiency of the rewind system, in that there would be less players etc to track?

In the past the common forum thought was we went to 12 vs 12 for two reasons- first because there was a push for bigger battles, but mostly it was thought it was a big reduction on the server load because fewer matches would be happening at once.

I am curious if the current player population numbers combined with more recent optimizations to HSR(like the bone tracking etc) would allow for an overall benefit to return to 8 vs 8. Things like matchmaker, but also other less code oriented design things like TTK and sheer amount of potential focus fire etc could be addressed.

[LordNothing]

i think the usually approach to collision detection is to start with the cheap culls and work your way down. like start with aabbs or collision spheres and work your way down to say complex systems of convex hulls (with animation). you can immedietely cull things that are a ways behind your or out of range of your guns or movement. if the fastest mech can only move 3 meters in a frame, why bother checking beyond 6 meters in front of you (of course throwing in some leeway for when your framerate drops and the time to move that frame increases as a result).

then lag throws a 5th dimension into the problem. il be luckey if i get my collision spheres working in my own engine.

Edited by LordNothing, 07 September 2016 - 10:14 PM.

[MrMadguy]

Biggest MWO's problem - is that it's hit detection system is very irresponsive. There is no way for player to know, if he deals full damage to target or just a tiny part of it. It's intuitive for player to assume, that if he sees crosshair flashing - then he deals full damage to target. And in most cases it's not true. Hitscan weapons can't be pinpointed for long enough period of time - you usually just scratch enemy's armor. Ballistic weapons suffer from convergence being broken via leading target. When player fires 2xGausses and sees flashing crosshair - he assumes, that both Gausses hit and deal full damage. But it isn't the case too. Converge completely brakes at close range - sometimes I accidentally hit Lights, that run in front of my arm, without even aiming at them.

Most other games have either: 1) Some kind of hit feedback system, that shows player, how much damage he really deal. This is the essence of controlling - you send some control signals and use feedback to validate this signals and correct them if it's necessary. Such system makes game more responsive. When you do something and don't even know, if you do it wrong or right - this is obviously a bad game design. 2) Some kind of shotting assistance system. I.e. we sacrifice some shotting precision to make game more intuitive and responsive.

Biggest problem of this game - it has many small fast moving targets. And it simply looks like game engine simply can't handle such high speeds and small sizes of hitboxes and it's very unstable as the result. Yeah, may be some assumptions and interpolations work well for slow moving targets, where error margin is smaller, than 'Mech size itself. But it isn't the case for Lights, that can have so high speed, that their per tick movement is 2x-5x longer, than their size. But may be it happen not due to hit detection and quantization problems, but due to simple ability of human to react so quick and to aim so precisely. I.e. PGI simply don't take ergonomics into account. Sometimes it may seem to you, that you've hit your target - and only slow replay can reveal, that you've missed. Human eye just can't catch so small detail. When some skills, needed for playing game, are beyond human capabilities - this usually isn't considered a "hard game" - it's simply considered a bad game design.

This problem causes big imbalance between 'Mech classes. Some 'Mech classes require you not only to become more experienced and skillful in game itself, but to also deal with restrictions of human nature. I think, it's wrong. When you play Light/Medium 'Mech - you need your skill only. Other 'Mechs also require very high level of reaction and aiming skills - sometimes physically impossible ones.

I know, that MWO is considered to be simulator - not arcade shooter. But there should be some compromise or middle ground between being precise simulator and good game. Simply because playing game - is two-way process. Game should respond to your actions. When it doesn't - may be it's good simulator, but bad game.

Matchmaker could solve this problem - it could match players with equal skills and physical capabilities only. But, as we all know, it doesn't work properly in your game, as it's based on W/L only - not on players' personal skill. It doesn't take 'Mech/variant/build into account - so it doesn't care, if I play top Meta 'Mech or some "just for fun" crappy one. That's why for example I play on such rating, where enemies' skill is far beyond my physical capabilities.

Map design contributes into problem too. This game is actually about dealing as much damage as possible, while taking as few damage, as possible. And cover - is major part of this game. That's why making maps with open centers or without any cover at all - is obviously bad game design. Your game is too snipe-biased. Long range weapons are always effective and don't have any downsides, while short range weapons have extremely limited use. May be more skilled players can deal with this problem somehow, but not me, sorry.

Conclusion? Your game is way too "Hardcore" - it isn't friendly to players with different physical capabilities and skill caps. You're either the best or nobody here - no middle ground. ESports only - no room for playing just for fun.

Edited by MrMadguy, 08 September 2016 - 03:18 AM.

[Kmieciu]

Neema Teymory, on 07 September 2016 - 10:21 AM, said:

The next biggest related system is movement. If design decided to suddenly add a game mechanic that allowed a mech to stop on a dime and turn in the opposite directly, this would have drastic consequences for the rewind implementation--even to the point of breaking it.

In case you haven`t noticed that already happened. Try testing LCT-1V (90% accel/decel quirks and XL190 engine). The trick is to use throttle decay because it unlocks mechs full acceleration (throttle bar goes instantly to 100%).

The interesting thing is, while this mech is almost invulnerable to lasers and hitscan weapon, occasionally you can get killed by a gauss slug.

Edited by Kmieciu, 08 September 2016 - 12:27 AM.

[Chados]

Explains why they are so reluctant to implement new weapons tech. To bring in the HAG 40 or some of the other 3056+ era weapons they'd have to program an awful lot of variables and it would get pricey quick.

[Greyhart]

well that makes my head hurt.

But this is the sort of interaction we should be getting more of. I know it takes time to do a response but a bit of explanation goes a long way.

interesting to know the problems caused by tracking hits.

[Kangarad]

so you need to remove the accel /decell quirks for the system to be functioning correctly? sounds great.

Edited by Kangarad, 08 September 2016 - 04:43 AM.