112 replies to this topic

[Idealsuspect]

For my part i have to say i never found that MWO was really weak about netcode and specially with lag compensation we are talking about....

• Using AC20 before and after velocity nerf ( this nerf did happend maybe 3 years ago ) well even after i was able to legg shoot a spider running while jumping at 500 meters range ( of course it was a part of luck but not only luck ).
  
• Now no real problems for give some ERPPC shoot to locust turning around me with only 15% velocity quirks. ( it's of course another thing when locust is doing forward-backward move on the edge of something )
  
• And as a ERLL user well yesterday ( again ) i did kill a SPL locust dancing around me with only 2erlls while my whole potatoes team was only able to spect... well after kill this locust i was almost dead but i talk about clan ERLLs also biggest duration weapon of this game against the faster and smaller light of the game ( PGI nerf it more plz ? )

Every goods pilots know this : The thing with all multiplayer games is that what you see in your client isn't the entiere true reality ( morpheus with neo )


>>> Also you have to adjust yours hits by compensate your ping, your target ping and the velocity or duration of your weapon with your potential futur position ( where shoots come from ) agaisnt your potential target's position ( where shoots have to be after duration/velocity prediction ).

All of this come with amount of time using the same weapon, experience or in fact routine/habbit if you prefer ...
If the pilot try to improve his shooting ability he can solve or smooth the lag compensation effect to his advantage... others pilots who don't even try ( i know sometime they can't coze hell's connection or potato computer FPS me i can't on oceanic servers so i simply don't play in oceanic server ) thoses guys will play a noskill streak boat or lrms boat till the end of this game be tier 1 and be happy whatever.


Ok:

• What see the server or in fact what the server predict = the reality.
  
• What your client show you = only a fake reality you have to deal with till server reality is updated to your client.
  
• What your client show you when you spect others pilots = how your client reality is fake

( And for most of us we don't notice bigs issues while specting others it proove that MWO job about lag compensation is ok/done/acceptable.. )



So i repeat MWO haven't huge problem with this coze we as player can predict or adapt yours shoots with lag compensation if we have noticable issue with hit reg...





**** But there is one thing really tough to adapt: it's ghost collision so i ask Neema for infos about this problem.
I dont know if this term is exact so i explain you what i have in mind.

Ghost collision:

• When a target hug you or you hug the target, you and the target can't move coze huging and 0.5 second ( after one alpha ) target disappear, you get a little teleportation ( or not ) and the target is behind you ( maybe the target got this ghost collision too but most of time the target is already shooting at you while you search it.. xD )

Neema Teymory, on 07 September 2016 - 10:40 AM, said:

Maybe one thing could be upgraded for improve a bit MWO about what we are talking ( server reality/client reality about real time mechs position) it's the tickets exchange rate between server and clients about mechs "real" positions for make ghost collision less visible/important.

Ghost collisions are really an issue which maybe could be solved with more exchanges between server and clients?
Or it's to much performance-eater because lag compensation system for weapon is linked with mechs position predictions ( and in fact all 8 components position prediction x24 players ) ...

A double layer with 2 differents speeds could be a possible way ?
One high ticks speed only about CT and Legs for improve mech position lag compensation and make ghost collision
and one another lower ticks speed about 8 components position for keep calculate weapon lag compensation like we have now ..
or it's already how it work ?
Or maybe i am totally wrong about why this ghost collision issue appear so big and impossible to predict/adapt during a match... Need some infos from someone who deal with this .




Ty to the OP for sharing gamasutra infos and ty to the PGI netcode slayer Neema for sharing with us too..

Edited by Idealsuspect, 09 September 2016 - 07:43 AM.

[MrMadguy]

Neema Teymory, on 08 September 2016 - 05:02 PM, said:

Hey TFun90,

For most of these videos, it seems like hit detection is actually accurate, but maybe the complaint is that the amount of damage being dealt is incorrect? It looks like for two cases there are actual problems though, the one with the Cataphract the first shot misses on a quick swing, and the jump jetting Jenner is missed with the small lasers. As I said before, the rewind algorithm isn't perfect, so there are cases where misses can occur. However, I'll make a note of these, and I'll try to open up some time to do some further investigation soon. In the meantime, if you have more cases, you can PM them to me. Thanks.

All you need to know: I don't know, why it happens, may be due to specific hit boxes, that can't be properly handled by server, due to some sorts of desyncs or something else, but some 'Mechs, specifically Lights ones, have long going history of having hitreg problems. Biggest offenders are Arctic Cheetah, Firestarter, Spider. Some other Light 'Mechs have some sort of unstable hitreg, when sometimes either you or your team just can't do any damage even to relatively slow Light, but sometimes you can one-shot them via shots, that are felt like purely accidental (without any precise aiming) - Locust, Raven, Jenner. Also using JJs decreases chance to be hit for them by approx 70%. I know, that using JJ causes more convergence problems for Ballistics. But Lasers have bad hitreg on "flying" targets too. So sometimes Light, that is constantly spamming JJs - is almost completely invulnerable. So since some moment I simple stopped to waste my firepower on "flying" Lights - I simply wait for them to land.

But Light problem - isn't the only one. Some other 'Mechs, that have relatively high speeds or ability to twist fast (Mediums mostly) - can also have "refuse to die" problems. It's when you shoot your 100% accurate Alpha into their cored red CT, when even a scratch should kill them, and...they simply refuse to die - they run with completely cored CT and refuse to die, till their STs won't be destroyed too. This is especially annoying, when other players are able to rip your ST in any situation and from any distance via 100% precise shots.

I agree, that I could have had hitreg problems, when I was playing on US servers from Europe and my ping was way too high - I needed to use unpredictable leading (when right leading angle is always different, so you just can't predict right one) even for lasers. But now my ping is around 45-50. But hitreg is still ******.

I don't know. Your HSR tech looks a little bit clunky and unreliable. If you can record, let's say, one second of game (30 past server frames) - then why can't you use some kind of "play in the past" tech? I.e. when player with high ping plays as normal (like if he would have 0 ping), but it the past phase of the game? I.e. when his projectiles use one of the past game states for hit detection, instead of current one? Virtually there should be no difference for server, whether frames are being received from players in real time or being played back from buffer - at the end they're just series of 'Mech positions. If you have hit detection algorithm, that can analyze current frame, then why can't this algorithm be redirected to one of the past ones?

Edited by MrMadguy, 09 September 2016 - 07:49 AM.

[Mechwarrior Buddah]

Neema Teymory, on 07 September 2016 - 10:21 AM, said:

Again, it's tough for me to say what all other games do, but I can give you a sense of the extra work MWO does in order to lag compensate weapons. When you fire, the server needs to rewind all 'eligible' targets--including friendly targets. This is because if you don't rewind all targets, you may strike a 'present' target and incorrectly detect a hit on the wrong target.

Is that how the HSR works and why its gotten better than it used to be?

This isnt meant as a criticism, but why cant we get more replies like yours in here? More interaction like this would go a loooong way towards putting out some of the dumpster fires that start up around here.

who says you never need math O.o

[Mechwarrior Buddah]

Neema Teymory, on 07 September 2016 - 10:21 AM, said:

A projectile rewind is this same rewind check plus a little more work. Instead of rewinding by ping, we need to rewind by an adjusted ping that is based on a line test of an adjusted velocity vector against each potential target's geometry. This is better explained in the Gamasutra article, so I won't repeat everything here. This amounts to 23 additional line checks against target geometry. We can't only use the bounding box of the target here, because we need an accurate hit time to get the correct adjusted ping value. Of course, the bounding box is used as a broad phase optimization.

So to summarize the extra work the server does for lag compensation:
- For a trace fire weapon: 23 line box tests + N (usually 1 or 2) target rewinds + 1 world ray trace
- For a projectile weapon: 23 line target tests + everything else a trace fire weapon does

Im curious, is it easier for IS or clan weapons? Or things like the lbx for that matter. Cause with the IS you only have the one projectile, but with the clan ones they autofire. Are the projectiles WE see on things like the clan autocannons real or is it a visual effect and they in fact only fire a fewer number of projectiles?

Sorry if these are silly questions, I was just really curious

Neema Teymory, on 07 September 2016 - 10:21 AM, said:

The next biggest related system is movement. If design decided to suddenly add a game mechanic that allowed a mech to stop on a dime and turn in the opposite directly, this would have drastic consequences for the rewind implementation--even to the point of breaking it.

like how mechs used to be able to do with jump jets? Is that why that was changed?

----look at me coming out of my bittervet bunker to ask questions

[Mechwarrior Buddah]

Zordicron, on 07 September 2016 - 09:40 PM, said:

Would reducing matches to 8 vs 8 have a positive impact on the efficiency of the rewind system, in that there would be less players etc to track?

good question

[MrMadguy]

Mechwarrior Buddah, on 09 September 2016 - 08:08 AM, said:

Is that how the HSR works and why its gotten better than it used to be?

This isnt meant as a criticism, but why cant we get more replies like yours in here? More interaction like this would go a loooong way towards putting out some of the dumpster fires that start up around here.

who says you never need math O.o

Yeah. There some problems, that can easily be fixed, but PGI simply need somebody to actually listen for feedback, instead of ignoring it. Examples:
1) At least make PSR per 'Mech - performance for different 'Mechs may be way too different. In my LRM60 boat I'm Tier 1 god, but in some obsolete IS Assault brawler - I'm Tier 5 player
2) Separate maps, that lack cover (can't you see, that players hate them?), into separate "Sniping" mode, so players would know, that long range weapons are mandatory for this mode
3) Stop messing Mountain Line camo on new 'Mechs! Leave it alone! I like old version of it! If you want to have more flag-like camo in your game - make separate one and call it "Flag".

[Mechwarrior Buddah]

MrMadguy, on 09 September 2016 - 08:22 AM, said:

Yeah. There some problems, that can easily be fixed, but PGI simply need somebody to actually listen for feedback

From what Ive seen THEY need to listen, but the hard thing about that is knowing WHO to listen TO (as pointed out to me by someone earlier)

and thats not an easy thing to determine. They NEED to listen to SOMEONE outside of their dev house though and badly

Edited by Mechwarrior Buddah, 09 September 2016 - 08:25 AM.

[Hunter Watzas]

Thanks for the awesome responses. I do work in state estimation and control for unmanned vehicles and so i was wondering if you guys employ any types of probability into the equations for rewinding to better estimate/track people's pings and perhaps even "user inputs".

I assume most rewind time scales are 100s of milliseconds or less so maybe its not needed as much with the assumed linearized models.

[1453 R]

Mad props to Neema for a nice and meaty under-the-hood discussion. It's awesome to see some pieces of how the game does what it does laid out; this sort of thing makes it a lot easier for players to grasp what may or may not be possible/viable feedback and how or why certain things happen the way they do. I did not know that warm-up (Gauss charge?) ideas would have a potentially very serious effect on HSR, actually. That's really valuable to know given how much I like the idea of warm-ups/charge timers for certain (potential) weapon types.

Keep it up, tracking with interest.

[MrMadguy]

Mechwarrior Buddah, on 09 September 2016 - 08:24 AM, said:

From what Ive seen THEY need to listen, but the hard thing about that is knowing WHO to listen TO (as pointed out to me by someone earlier)

and thats not an easy thing to determine. They NEED to listen to SOMEONE outside of their dev house though and badly

Yeah. You shouldn't listen to players, when they have mutually exclusive interests - some ask for nerfs, but others ask for buffs. But there are universal problems, all players are affected by. Terrible hitreg - is one of them. But there are some others. You just need person, who will sort this problems out - who will determine, what is good for players (good for one player, but bad for another) and what is good for game (i.e. for all players at once, even if some of them will become sad as the result - fixing balance for example).

I don't know. I need to admit, that PGI has decent artists and great system developers. But there is thing, they really need, that makes difference between them and, let's say, Blizzard - great game designer! I.e. person, who simply knows, what to do to make game fun for as many players, as possible. Currently it seems, like PGI don't even know, what to do to make game better. They simply follow "Does it work? New 'Mechs are being sold? Don't touch anything!" kind of mentality.

Edited by MrMadguy, 09 September 2016 - 09:02 AM.

[Neema Teymory]

Hey everyone,

I appreciate all the interest in this topic. It's getting hard to keep up with all the questions. I'm always very busy around here, but I'll do my best to continue answering. Please don't take offense if I skip you, some of your questions, or am light on details. If you feel ignored and you really want a question answered, feel free to PM me and I'll try to answer your question the best I can.

Idealsuspect, on 09 September 2016 - 07:08 AM, said:

Maybe one thing could be upgraded for improve a bit MWO about what we are talking ( server reality/client reality about real time mechs position) it's the tickets exchange rate between server and clients about mechs "real" positions for make ghost collision less visible/important.

I mentioned earlier that increasing tick rate has an unfortunate consequence on bandwidth usage for the server. It is true that tick rate will have an effect on collision, but increasing tick rate alone will not make this problem any better, and the resulting increase in bandwidth will certainly cause a whole host of new and worse problems. Mech on mech collision is really a problem in the game, and we are fully aware of it. These collisions are not handled correctly on the server at all, which is why you end up with such random outcomes when Mechs run into each other. This is something that we definitely have wanted to improve since before I can remember, but it has never become a high enough priority item to commit time to.

MrMadguy, on 09 September 2016 - 07:38 AM, said:

But Light problem - isn't the only one. Some other 'Mechs, that have relatively high speeds or ability to twist fast (Mediums mostly) - can also have "refuse to die" problems. It's when you shoot your 100% accurate Alpha into their cored red CT, when even a scratch should kill them, and...they simply refuse to die - they run with completely cored CT and refuse to die, till their STs won't be destroyed too. This is especially annoying, when other players are able to rip your ST in any situation and from any distance via 100% precise shots.

As I mentioned earlier that there was a bug with invulnerable CTs on certain Mechs. It was a bug related to weapon doors that was fixed last patch (I believe?). It sounds like in some cases you may have been experiencing this bug. However, if you still experience it, please report the problems, and if you happen to have video that would also be great. As I mentioned earlier, if/when some time can be freed up, I would like to dig into more of these specific cases to find out what is going wrong.

Mechwarrior Buddah, on 09 September 2016 - 08:13 AM, said:

Im curious, is it easier for IS or clan weapons? Or things like the lbx for that matter. Cause with the IS you only have the one projectile, but with the clan ones they autofire.

Each projectile is treated individually at the time it is fired, so they can each be treated as isolated cases. So, it doesn't really matter how many are fired, except for that there will be more work to do (this is a bit of a over simplification, but its true enough. Since, a projectile fire delayed from the time input is received has consequences on how we calculate the player's ping).

Mechwarrior Buddah, on 09 September 2016 - 08:13 AM, said:

Are the projectiles WE see on things like the clan autocannons real or is it a visual effect and they in fact only fire a fewer number of projectiles? Sorry if these are silly questions, I was just really curious

The projectiles you see are indeed the projectiles that will spawn on the server as well. For certain weapons, projectiles are grouped into something called volleys that allow use to limit how much information needs to be transmitted between the client/server.

MrMadguy, on 09 September 2016 - 08:22 AM, said:

Yeah. There some problems, that can easily be fixed, but PGI simply need somebody to actually listen for feedback, instead of ignoring it. Examples: <snip>

Hey MrMadguy, I can understand your frustration on these various issues. Unfortunately, these things are not within my control to change. PSR was a system our design team came up with after they felt that the previous ELO system was a failure. PSR on the whole has been better received than the previous matchmaking system, and as such we feel it has been a success. That is not to say it is perfect. In general, for these types of complaints or specific feedback you can direct them to our CSR team, they compile them and direct them to the appropriate department. That doesn't guarantee they will be changed, but at least it raises the concern.

Hunter Watzas, on 09 September 2016 - 08:39 AM, said:

Thanks for the awesome responses. I do work in state estimation and control for unmanned vehicles and so i was wondering if you guys employ any types of probability into the equations for rewinding to better estimate/track people's pings

We don't use probability when calculating. In fact, calling it a "ping calculation" is a bit deceptive. The world movement state is actually timestamped. So when the client fires, they let the server know which timestamp they were firing at and an "inter-world state delta" to account for interpolation between world states. When the server receives the request, using that timestamp and the delta the server knows exactly where to rewind to.

Hunter Watzas, on 09 September 2016 - 08:39 AM, said:

and perhaps even "user inputs"

The server only "predicts" (I use this term loosely because the prediction of the server ultimately becomes the reality) an input from the client if they have not sent any input for a long enough time. If such an input arrives after the server has already predicted it, the client will need to be corrected. This is important to prevent a certain type of cheating called "slow hacking". If the server did no "prediction" for the client's input, a client can effectively pause their simulation on the server by stopping their input stream. This kind of cheating isn't quite as "glamorous" as speed hacking, but there are still some useful applications--you could stop instantaneously by pausing your input stream when you know a projectile is going to intersect your path and you can't stop in time or just constantly pause and resume your input stream to make your motion so erratic that its just too hard to hit you. This is not possible in MWO.

[Navid A1]

Thank you Neema for answering our questions.


Here is my question:

As the road map suggests, Real time IK is going to make a comeback into the game. it means that the leg positions won't be according to pre-keyframed animation sequence... but based on real-time ray tracing (or any other similar mechanics).

Now, based on how HSR works (rewinding mech positions and animation), Will IK pose any problem when you want to rewind a mech state?
Because IK is done in real time and is not stored as a frame number as it is now.

[Neema Teymory]

Navid A1, on 09 September 2016 - 12:43 PM, said:

Thank you Neema for answering our questions.


Here is my question:

As the road map suggests, Real time IK is going to make a comeback into the game. it means that the leg positions won't be according to pre-keyframed animation sequence... but based on real-time ray tracing (or any other similar mechanics).

Now, based on how HSR works (rewinding mech positions and animation), Will IK pose any problem when you want to rewind a mech state?
Because IK is done in real time and is not stored as a frame number as it is now.

This is a good question. The way we "rewind" the animation is not by playing the animation backwards or to a specific key frame. That's too complicated. Instead, after each movement update, we capture the position of all the parts of the animated mech, and save them along with the other rewind state information like position, rotation, etc. When it's time to rewind the Mech, not only do we restore the position, rotation, etc of the Mech, we also restore the position, rotation, of all the animated parts.

IK will run on the server as well. So, as long as positions/rotations are synchronized between the client and server, the IK calculations should result in the same placement of legs. However, this does introduce a new possibility for desync in the legs. In practice, our testing has shown this shouldn't be a problem.

To give you an idea of when this might be a problem: imagine a mech standing at the edge of wall, roughly shin height. Depending on the mech's orientation, IK will try to lift the leg of the mech onto the edge of the wall, or leave the leg on the ground. At some point the leg is either going to IK on top of the wall or not. There is no intermediate state. This edge is discontinuous in the motion of the leg, which means even a tiny amount of error in the position or rotation of the Mech can result in a large difference in leg placement. Namely, for the client the leg may be on the wall, and on the server it may not be. As I mentioned, in practice this shouldn't be a problem. If a target were to stop moving, your simulation of that target converges with the server, so there should be no error. And if you were standing still next to a wall, you are probably a sitting duck, so this is already an uncommon case. So, this is really a tradeoff, and we have decided that the visual fidelity is worth the minor possibility for errors, that in practice shouldn't have a negative impact to the game.

[Navid A1]

Thanks for the detailed info regarding the current rewind mechanics and how it is going to be for real time IK.

I've been asking for the the return of real-time IK since 2014 and it is great we're gonna see it back in the game finally.
I'd totally accept minimal transient edge case errors trade off for that!

Edited by Navid A1, 09 September 2016 - 02:29 PM.

[Navid A1]

Another question.

Sometimes the cockpit shake animation kicks in even when there was no hit.

Most clear when someone misses a close shot with an AC20 or a gauss rifle.
Is it something that is expected to happen?

Edited by Navid A1, 09 September 2016 - 03:27 PM.

[Neema Teymory]

Navid A1, on 09 September 2016 - 03:27 PM, said:

Another question.

Sometimes the cockpit shake animation kicks in even when there was no hit.

Most clear when someone misses a close shot with an AC20 or a gauss rifle.
Is it something that is expected to happen?

The cockpit shake is predicted locally. The server doesn't tell the client when the cockpit should shake. So when there is a misprediction, the shake may occur when no hit occurs on the server or vice versa. This is technically a bug. The cockpit shake should probably be server controlled, and it was something we were aware of when we made the projectile explosion effects server controlled. But since there weren't complaints about it, and because we wanted to save on bandwidth, we decided not to control this on the server. It's something in my backlog, and I would like to revisit this at some point.

[nitra]

Just want to say thank you for taking the time to answer Q's it refreshing to see some detailed dev responses in regards to the finer details of MWO.

[Y E O N N E]

This thread is an amazing info-dump, thank you PGI staff for the interaction!

[MrMadguy]

Neema Teymory, on 09 September 2016 - 12:30 PM, said:

As I mentioned earlier that there was a bug with invulnerable CTs on certain Mechs. It was a bug related to weapon doors that was fixed last patch (I believe?). It sounds like in some cases you may have been experiencing this bug. However, if you still experience it, please report the problems, and if you happen to have video that would also be great. As I mentioned earlier, if/when some time can be freed up, I would like to dig into more of these specific cases to find out what is going wrong.

Do you mean, it can happen with 'Mech, that has weapon doors only? I ask it, cuz I experienced this problem with many different 'Mechs. Quickdraw for example. I guess, there have been moments, when I have been invulnerable too - in my Firebrand for example. Also sometimes it seems, that problem is caused by some sort of desync. I.e. you aim at CT, but hit arms instead. And this can happen even with huge slow 'Mechs, such as Atlas for example. Also if this problem occurs, it occurs for all 'Mechs in this match - i.e. not only I experience such problems, but it seems, that at least all 'Mechs in my team experience them too. Funny thing, but when all 'Mechs are affected by this problem, game feels much more enjoyable due to much higher TTK.

Edited by MrMadguy, 09 September 2016 - 11:28 PM.

[El Bandito]

MrMadguy, on 09 September 2016 - 11:16 PM, said:

Funny thing, but when all 'Mechs are affected by this problem, game feels much more enjoyable due to much lower TTK.

Higher.