112 replies to this topic

[nehebkau]

Thanks so much for posting this @OP. Really enjoyed the read!

[Neema Teymory]

Rebas Kradd, on 07 September 2016 - 08:48 PM, said:

Is pure server-side authority a unique choice amongst CryEngine games, or do other games do this?

I think what you are asking is the choice to not trust the client unique to CryEngine games or not. The answer to that is no. If you are developing a client-server multiplayer game, the decision of whether to trust the client or not has nothing to do with choice of game engine. In fact, each gameplay feature in a multiplayer game could be implemented differently. It just depends on the requirements of your game. For example, vision mode changes in MWO are trusted by the client (sometimes we say not server authoritative). The client can change their vision modes whenever they want, and just relay what they did to the server. The reason for this is that there are no rules around usage of vision modes that need to be enforced. If there were rules though, i.e. you needed a certain module. Then this would be something that we would want to check on the server to avoid cheating. i.e. we would no longer be trusting the client in this case. For us, it was very important to prevent any kind of cheating related to hit detection, so we decided early on not to trust the client at all. We don't trust the client's movement inputs either (the server does all kinds of verification on your inputs to ensure they make sense), and various other things. But as you can see, each feature could be, and in some cases is, implemented differently.

[Neema Teymory]

Zordicron, on 07 September 2016 - 09:40 PM, said:

Would reducing matches to 8 vs 8 have a positive impact on the efficiency of the rewind system, in that there would be less players etc to track?

If there were less targets, certainly there would be less work to do. But as I mentioned, the server is optimized to limit the amount of work to as few relevant targets as possible, so it manages 12v12 fine and reducing to 8v8 should have a negligible impact to the rewind system.

Zordicron, on 07 September 2016 - 09:40 PM, said:

In the past the common forum thought was we went to 12 vs 12 for two reasons- first because there was a push for bigger battles, but mostly it was thought it was a big reduction on the server load because fewer matches would be happening at once. I am curious if the current player population numbers combined with more recent optimizations to HSR(like the bone tracking etc) would allow for an overall benefit to return to 8 vs 8. Things like matchmaker, but also other less code oriented design things like TTK and sheer amount of potential focus fire etc could be addressed.

A lot of these comments fall under the category of design, so I can't speak much to that. But from a technical standpoint, it should be no mystery that an 8v8 match is certainly more manageable from a server perspective. There would be less bandwidth used and fewer calculations necessary to do, both in terms of rewind, and general gameplay processing.

[KodiakGW]

Neema Teymory, on 08 September 2016 - 09:27 AM, said:

...reducing to 8v8 should have a negligible impact to the rewind system.

First off, thank you for replying to everyone's questions regarding this subject.

Is is possible, after the PTS is done with Energy Draw testing, that we can set it up to test 8v8 hit registration/rewind system testing to confirm your theory? I'm asking because I have seen hit registration not be the same from one game to the next. I don't record all my games, but some of the ones I have posted show some questionable hit registration at one point or another. Plus, you have probably hear many complaints about "broken hit boxes" that might be attributed to hit registration.

[Neema Teymory]

KodiakGW, on 08 September 2016 - 11:48 AM, said:

First off, thank you for replying to everyone's questions regarding this subject.

Is is possible, after the PTS is done with Energy Draw testing, that we can set it up to test 8v8 hit registration/rewind system testing to confirm your theory? I'm asking because I have seen hit registration not be the same from one game to the next. I don't record all my games, but some of the ones I have posted show some questionable hit registration at one point or another. Plus, you have probably hear many complaints about "broken hit boxes" that might be attributed to hit registration.

Hey KodiakGW,

What goes on PTS is not something in my control. It's generally, something used by design to test new features. You could ask the design team if they are willing to do something like this. But if I recall correctly, PTS is already booked for something new after Energy Draw testing is done.

The rewind system isn't perfect, certain conditions may cause false positives or negatives, specifically variations in ping, or "jitter" can be problematic. With that said, I usually encourage users to send me reports of hit registration problems via PM, especially if you have video. It is always possible there is some kind of bug lurking in the code, so I am always open to investigating (when there is time )

[Pariah Devalis]

Neema Teymory, on 08 September 2016 - 12:00 PM, said:

Hey KodiakGW,

What goes on PTS is not something in my control. It's generally, something used by design to test new features. You could ask the design team if they are willing to do something like this. But if I recall correctly, PTS is already booked for something new after Energy Draw testing is done.

The rewind system isn't perfect, certain conditions may cause false positives or negatives, specifically variations in ping, or "jitter" can be problematic. With that said, I usually encourage users to send me reports of hit registration problems via PM, especially if you have video. It is always possible there is some kind of bug lurking in the code, so I am always open to investigating (when there is time )

Relating to that, given how many people claim that the PPC family of weapon systems seem to have a disproportionate number of registration errors, is there anything unique about them that might lead to a decreased detection rate? Or, I suppose a more fair question to ask would be if your data supports or denies such claims in the first place?

[Quicksilver Aberration]

Pariah Devalis, on 08 September 2016 - 12:03 PM, said:

Relating to that, given how many people claim that the PPC family of weapon systems seem to have a disproportionate number of registration errors, is there anything unique about them that might lead to a decreased detection rate?

It's not just PPCs, all projectiles have this to a degree. I've seen me and another Kodiak put 3 (PPC/Gauss) alphas into a Mauler before the 4th shot between the two of us finally registered (keeping in mind it was standing still). SRMs also have this problem especially with splat Jenners.

Edited by Quicksilver Kalasa, 08 September 2016 - 12:07 PM.

[Tibbnak]

Does the server use dynamic tick rates? If so, does the server scale the prediction to compensate for the loss of granularity when it goes down under load? Might be a contributing factor.

[LordNothing]

Chados, on 08 September 2016 - 03:47 AM, said:

Explains why they are so reluctant to implement new weapons tech. To bring in the HAG 40 or some of the other 3056+ era weapons they'd have to program an awful lot of variables and it would get pricey quick.

thats what structs, classes and vectors are for.

[LordNothing]

i like what im seeing in this thread. i love it when the game devs get down and dirty with these kinds of technical discussions. in any given game community there is always a subset who are obsessed with the technology moreso than the gameplay itself, and we just feed on this kind of stuff. its why im making a game engine, not because i want to make a game, but because i like to know how things work (and why i refuese to solve problems with '#include <something.h>', unless of course i happened to write something.h myself).

Edited by LordNothing, 08 September 2016 - 12:22 PM.

[Neema Teymory]

Pariah Devalis, on 08 September 2016 - 12:03 PM, said:

Relating to that, given how many people claim that the PPC family of weapon systems seem to have a disproportionate number of registration errors, is there anything unique about them that might lead to a decreased detection rate? Or, I suppose a more fair question to ask would be if your data supports or denies such claims in the first place?

PPCs are handled the same way as any other projectile, however, they have different stats, such as speed.

Quicksilver Kalasa, on 08 September 2016 - 12:07 PM, said:

It's not just PPCs, all projectiles have this to a degree. I've seen me and another Kodiak put 3 (PPC/Gauss) alphas into a Mauler before the 4th shot between the two of us finally registered (keeping in mind it was standing still). SRMs also have this problem especially with splat Jenners.

This definitely sounds like a bug, if you have videos demonstrating the problem, send me a PM and we can look into it. Sometimes these problems aren't hit detection related, but damage calculation related. For example, you will probably recall recently there was a bug with Mechs that had multiple weapon doors (I think the Kintaro specifically) where when the doors were closed (or open? don't remember exactly) the associated component became invincible. This was simply a bug in the code that calculated the doors giving the components 100% damage reduction. It wasn't related to hit detection.

[Neema Teymory]

Tibbnak, on 08 September 2016 - 12:12 PM, said:

Does the server use dynamic tick rates? If so, does the server scale the prediction to compensate for the loss of granularity when it goes down under load? Might be a contributing factor.

Without getting into too much nitty gritty, the server uses a fixed tick rate. However, if for some reason the host server experiences heavier load, it's possible that a specific game instance may not be able to sustain it's fixed tick rate. This happens very rarely. There have been instances where some hardware has failed on us, and subsequently the load spills over onto other servers--this can cause that kind of problem.

The server doesn't do any prediction, only the client does that. But rewinding is related to the communication latency between the client and the server. If the server begins to slow down below its fixed tick rate, this effectively increases the latency between the client and server. In this case, the position the server rewinds targets too will change appropriately. In theory everything should continue to work correctly, but in practice, this isn't what happens because the additional latency ends up being "jittery" due to the unpredictable nature of the slow down. But like I said, this happens very rarely and probably wouldn't be the source of a consistent problem.

I hope that answers your questions.

Again, if there is anybody out there with videos demonstrating problems, please report them. Feel free to PM me directly. We will investigate.

[LordNothing]

you know i find the narc to be even worse than ppc in reguards to hit registration. i noticed a lot of this durring the event., trying to get my narc kills. with ppcs hit failures are a 1/25 occurance and generally are not enough to get my goat, but with narcs its more like 1 in 5. i hit a lone mad dog, standing completely still dead center ct at point blank range and nothing (and no ecm disruption). after 3 narcs one finally stuck.

are these weapons using sphere colliders or is it just a line test? i thought about using a capsule collider with the nodes (center of where the hemisphere endcap connects with the cylinder) representing where the weapon projectile was in the current frame and the last, and if one of them intersects an objects bbox then a collision is likely. if i want more precision perhaps break it down into sphere colliders each one representing a different dt value to get sub-frame resolution.

Edited by LordNothing, 08 September 2016 - 12:46 PM.

[Quicksilver Aberration]

Neema Teymory, on 08 September 2016 - 12:38 PM, said:

If the server begins to slow down below its fixed tick rate, this effectively increases the latency between the client and server.

Any chance we will ever see an increased tick rate or whether that will help stability at all regarding predictions for short burst weapons like lasers and UACs which can often seem like they aren't doing enough damage I'm guessing due to fluctuations with connection during that short time period? I know I often have had to lag shoot (aiming for where someone might be according to the server instead of where they are on the screen) with lasers lately to get any decent hit registration for fast moving mechs.



I do wish I got the video of the myself and the other Kodiak putting 3 alphas into a mech without damage, but sadly I did not get that video, but luckily I haven't had that issue before, that was probably a one time sort of occurrence.

Edited by Quicksilver Kalasa, 08 September 2016 - 12:49 PM.

[Mawai]

I just wanted to thank the developer immensely for responding on the forums.

It is WONDERFUL to see someone from PGI engaging with the community.

[TFun90]

Shitreg clips, you desire?

Shooter-side terrain change/accelerations shitreg https://embed.gyazo....9be08550506.mp4

Collision/JJ/terrain shitreg:

Disappearing alpha (target was already damaged) https://embed.gyazo....58c5ccb5c56.mp4

Terrain desync shitreg: http://i.imgur.com/72AqVTO.gifv

Again: https://i.gyazo.com/...734c01e6480.mp4

Post-collision shitreg:

Lag leading a SDR clip: https://imgur.com/gallery/XTfuWOs


FYI this is across 2 computers, 3 California broadband internet connections wired and wifi. I would say that server vs sprite desync during acceleration changes caused by JJs/landing/terrain adjustment are the leading cause of disappearing shots. It's almost as if the prediction model weren't updated to account for all the quirks, and the terrain deceleration patch. I'm mostly okay with mech collision shitreg, it's just a shame when you open fire right as a target warps.

Edited by TFun90, 08 September 2016 - 03:14 PM.

[Neema Teymory]

Quicksilver Kalasa, on 08 September 2016 - 12:46 PM, said:

Any chance we will ever see an increased tick rate or whether that will help stability at all regarding predictions for short burst weapons like lasers and UACs which can often seem like they aren't doing enough damage I'm guessing due to fluctuations with connection during that short time period? I know I often have had to lag shoot (aiming for where someone might be according to the server instead of where they are on the screen) with lasers lately to get any decent hit registration for fast moving mechs. I do wish I got the video of the myself and the other Kodiak putting 3 alphas into a mech without damage, but sadly I did not get that video, but luckily I haven't had that issue before, that was probably a one time sort of occurrence.

Hey Quicksilver,

There is no plan to change the server's tick rate. The problem with increasing server tick rate is that it has a huge impact on bandwidth usage. Tick rate shouldn't have anything to do with the way damage is applied for UACs or any projectile in general. Laser damage is done over time, so it makes sense to link this to tick rate, but there is special code that deals with this situation on the server, and it should be dealing the correct damage. This is something you should be able to test in private matches. Of course, there can always be bugs

TFun90, on 08 September 2016 - 03:09 PM, said:

<snip> clips, you desire?

Hey TFun90,

For most of these videos, it seems like hit detection is actually accurate, but maybe the complaint is that the amount of damage being dealt is incorrect? It looks like for two cases there are actual problems though, the one with the Cataphract the first shot misses on a quick swing, and the jump jetting Jenner is missed with the small lasers. As I said before, the rewind algorithm isn't perfect, so there are cases where misses can occur. However, I'll make a note of these, and I'll try to open up some time to do some further investigation soon. In the meantime, if you have more cases, you can PM them to me. Thanks.

[Quicksilver Aberration]

Neema Teymory, on 08 September 2016 - 05:02 PM, said:

This is something you should be able to test in private matches.

Private matches actually tend to be the worst with hit registration, whether that has to do with the netcode, collision detection in general, or resolution is another story. It is easiest to blame netcode because training grounds doesn't necessarily have these same problems.

Lasers are especially finicky, for example take a mech jumping in the air, once you let off the jump jets, if you fire your lasers, your shots won't register against your target half of the time, especially if you are moving laterally at the same time with a decent speed. If I had videos I would show you but I don't normally record my play (probably should be unlazy at some point).

Neema Teymory, on 08 September 2016 - 05:02 PM, said:

For most of these videos, it seems like hit detection is actually accurate, but maybe the complaint is that the amount of damage being dealt is incorrect?

Correct, the damage being dealt doesn't line up with what it should be.

Edited by Quicksilver Kalasa, 08 September 2016 - 05:09 PM.

[TFun90]

Neema Teymory, on 08 September 2016 - 05:02 PM, said:

Hey TFun90,

For most of these videos, it seems like hit detection is actually accurate, but maybe the complaint is that the amount of damage being dealt is incorrect? It looks like for two cases there are actual problems though, the one with the Cataphract the first shot misses on a quick swing, and the jump jetting Jenner is missed with the small lasers. As I said before, the rewind algorithm isn't perfect, so there are cases where misses can occur. However, I'll make a note of these, and I'll try to open up some time to do some further investigation soon. In the meantime, if you have more cases, you can PM them to me. Thanks.

Hitreg is reasonably close on the QKD and HBK vs Jenner, and completely ****** in the other 4. A 30 damage 6MLas alpha should kill the Cheetah, whereas the result was a 1-2 damage armor flash. I intentionally lag lead the nonevasive spider after the first shot did squat, and still pop the tailing side torso which the HITSCAN cursor never even touched. A full flight of SRMs explodes on an enemy Jenner, and the following armor read is nothing below yellow. The Locust should have been smoked half way through the beam, not someone else's kill. Accurate hit box flashes != accurate damage registration.

Edited by TFun90, 08 September 2016 - 06:34 PM.

[Mawai]

Neema Teymory, on 08 September 2016 - 05:02 PM, said:

Hey TFun90,

For most of these videos, it seems like hit detection is actually accurate, but maybe the complaint is that the amount of damage being dealt is incorrect? It looks like for two cases there are actual problems though, the one with the Cataphract the first shot misses on a quick swing, and the jump jetting Jenner is missed with the small lasers. As I said before, the rewind algorithm isn't perfect, so there are cases where misses can occur. However, I'll make a note of these, and I'll try to open up some time to do some further investigation soon. In the meantime, if you have more cases, you can PM them to me. Thanks.

Just a quick comment. I also watched the videos and there are examples of both missed hit registration and possibly damage not being applied.

Hit registration seems to be not working for the small lasers missing the jump jetting Jenner IIC (which you mentioned) where no damage appears to be applied to the rear despite what looks like 4 or 5 small laser volleys that should have been sufficient to kill the mech ... none of the damage appeared to register.

In the first video ... six IS medium lasers are fired at a red/orange armored leg of an Arctic Cheetah that is standing still. All lasers appear to hit and the reticle turns red also indicating a hit. However, 6 IS medium lasers is about 30 damage which should have been enough to easily strip the rest of the armor and possibly kill the Arctic Cheetah by destroying the single leg remaining ... but that didn't happen. By the end of the video, little if any of the damage that should have been applied appears to have been registered. The video extended for almost 2 seconds after the end of the IS medium lasers firing. If the damage was going to register on the target then it should have done so by then even if the pings involved were 500ms.

I would think that the first video is an indication of a damage application bug ... or possibly an issue with the Arctic Cheetah geometry where a significant part (but not all) of the beam duration may not be registering as a hit for some reason.

Edited by Mawai, 09 September 2016 - 06:12 AM.