213 replies to this topic

[Stoicblitzer]

what the **** is going on? it just started happening to me. looked like i was going to center of map and then i saw out of bounds and died. jesus christ.

holy ****.

[K o i l]

Just want to add my name to the list.

Same deal...out of sync. Can move, fire with no issue but suddenly told I'm OOB and destroyed. Delayed chat, ppl see me as running into a wall, etc.

Glad it's being looked into, but weird how it occurred, as I was playing earlier.

I DID update my avast av software. When re-install completes, I will see if that fixed it, if not, will check and see if avast is doing anything funky, otherwise, I'll await Karl's hotfix.

[StealthAssassin]

This just started happening to me today also. I was playing fine yesterday. Now, suddenly without any client-side changes, my game seems to be out of sync. Consistent with other players reports in this thread, telltale signs of the problem include:

-Extremely delayed zoom ('z' has no apparent effect)
-Extremely delayed targeting ('r' has no apparent effect)
-Can run around freely, but other mechs seem to oddly walk through you, and weapon hits on screen result in zero damage
-Enemy team does not see you (likely because you are elsewhere on the map according to the server)

This is unplayable, and needs to be addressed immediately by PGI.

[TheMagician]

As I posted elsewhere, this started for me at approximately 9:12pm EST.

[K o i l]

Karl Berg, on 22 April 2013 - 01:38 PM, said:

Some of these changes actually went live a couple patches ago, and involved sending lots of small little acknowledgements back to the server so that the server could intelligently compress what it sends to your client. Unfortunately, once again we appear to have tripped some unfortunate behaviour in the engine which we are currently trying to track down and correct.

So, are these network optimizations the reason we've seen the slight warping, sudden re-orientation, "getting bumped but no mech near you" effect, etc during normal game play?

Edited by K o i l, 22 April 2013 - 07:43 PM.

[Kahoumono]

I have been off for a couple days and have play 4 games so far with a 50% out of sync rate.

[grayson marik]

Same here, stopped playing for now as it gets worse every day.

[DeaconW]

p00k, on 22 April 2013 - 06:16 PM, said:

same problem described here. first drop of the night, and i'm doing my startup sequence while my team walks off towards the battle. can't zoom in, can't target mechs, i can see people moving around and shooting, and i can shoot at them but it never registers any damage.

I FRAPed this occuring: http://mwomercs.com/...ost__p__2280128

[meteorol]

Having similar issues since two days.

[cyberFluke]

This will probably explain why half my hits against stationary assault mechs don't register. I gave up on the tourney at the weekend because of it. I got fed up of dying to noobs standing still unloading PPCs but not taking any damage when fired upon.

[pesco]

I'd like to know how hard PGI devs curse the decision to use a single player engine for a multiplayer shooter.

Edited by pesco, 23 April 2013 - 04:38 AM.

[AlmightyAeng]

Thanks Karl. It's very reassuring to hear from someone on the dev team 'Hey, we did something, we're aware there is a problem, and we are currently working on it.' #anythingisbetterthansilence

Thank you.

[Rixsaw]

I am experiencing this as well. I initially thought it was DirectX9.0c files that had been overwritten and needed to be re-installed. Tried to find the 9.0c installer instead of the web installer which just says you already have it installed.

I was on vacation last week, and logged in last night, and patched. When done with patching I joined a group and dropped into an 8 man. I noticed my zoom was delayed, advanced zoom didn't work at all. I moved into position fired on mechs with no red reticle confirmation of hits. Then noticed the enemy team didn't see me. At the end of the match my mech exploded part by part, while I was not being fired on. And it showed I had done zero dammage.

Based on what I was seeing, it was as if anything that required the server to receive data from me and respond with an acknowlegement was not working. Being a network engineer, I think I know what is broken.

You introduced a compression system, which streams some info into a buffer and then compresses it, and sends it to the server. The server receives these packets in order, and decompresses them. The decompression key may be in a seperate packet, and if the data packet arrives before the key the server can't plug that data into the stream to process it, so it is discarded. Some internet providers between my location and your server have multple links with various levels of load, so packets get load balanced over one link or the other an then sent. They arrive at the server out of order. You have to implement net code that has a 100-150 ms buffer, to allow for packets that arrive out of order to be re-ordered. You also have to have the compression key in every packet so that a single dropped or out of order packet doesn't mean a huge amount of input data is dropped or un-usable.

In addition to those changes listed here, you will also need to implement a max segment size for your data packets. It isn't always the case that packets will be able to grow up to a full 1500 bytes. You should limit your packet sizes to 900 bytes to be ultra safe. You might even have a netcode optimizer that runs at the start of MWO which tests the connection to the server and multi-pathing, as well as max segment size to optmize the client side data sending. In that way each client will be optimized for its current connection, clients with low latency and no packet loss or multi-pathing can use the best solution, and those with high latency and packet loss can use a less optimized approach.

Edited by Rixsaw, 23 April 2013 - 08:07 AM.

[Joker Two]

Thanks for the communication from the Devs. I haven't had this problem personally, but several members of my unit have.

Ghost Badger, on 23 April 2013 - 07:19 AM, said:

anythingisbetterthansilence

Edited by Joker Two, 23 April 2013 - 08:14 AM.

[The Bad Charlie]

There be hope!!!

[ElLocoMarko]

And the problem which was rampant Friday night has disappeared for me.
I find it odd that it is like a roving jynx that plagues you and then later spares you to strike someone else.

Good to hear that a fix is imminent.

[Divine Madcat]

One more victim signing in. Didn't know about this bug till I ran into it twice last night. "Fought" a whole match, only to realize I got no hits..

Next round did it right off the bat.. I quit the match and logged off for the night. What a joke PGI..

[Karl Berg]

K o i l, on 22 April 2013 - 07:43 PM, said:

So, are these network optimizations the reason we've seen the slight warping, sudden re-orientation, "getting bumped but no mech near you" effect, etc during normal game play?

Maybe; the failure cases for this issue are usually much more extreme, but please let us know if the hotfix makes any difference.

pesco, on 23 April 2013 - 04:37 AM, said:

I'd like to know how hard PGI devs curse the decision to use a single player engine for a multiplayer shooter.

... no comment

@Rixsaw

Not quite. Here is what was done:

Mech input state is queued up into a 20 hz stream of traffic sent from your system to the server. The server processes this state and relays it to all other clients. Obviously then, for a 16 player game, you're sending one set of inputs up to the server, and receiving inputs for 15 other players. For a 24 player game this is compounded, one set of inputs up and 23 sets of inputs down. This movement state traffic dominates all the other game traffic being sent in terms of bandwidth costs.

Taking a look at what is actually in that mech input state, you have some aim angles, throttle settings, jump jet status, torso turn settings, and a small collection of other essential states. At 20 hz, a lot of that state doesn't change from one input to the next, so it's a fairly reasonable optimization to only send input values when they change, commonly referred to as delta compression. Most of our optimizations are focussed on the data being sent to you from the server, since that grows with player count.

To start, we add a sequence id to map transmitted movement state to a specific identifier. Now every move we send to the client is tagged with its sequence number. If this movement packet requires previous state to decompress, we add that base state identifier as well and delta compress the current state against that base state before transmission.

Well, we're using UDP, and all this traffic is unreliable and unordered. The underlying movement system is set up in such a way that it will reorder or simply reconstruct lost traffic over a very small window of time. If any received input is too old it's simply discarded.

We still have to deal with the problem of knowing what states the client has received, so for every state the client receives, it send back a really small 'state ack' packet containing a small identifier and the last received sequence id. The CryNetwork layer handles combining all these little tiny packets into optimally sized packets for transmission for us. Now on the server it's quite simple to always delta compress against the most recently ack'd state for each client.

It's the transmission of these ack packets in combination with small levels of packet loss that have messed things up. My guess is that our sending of lots of tiny little messages is incorrectly triggering flow control logic in the network layer, but it will take some digging to really track down where and why this is happening.

Small update on the hotfix, QA has it and is testing it out in the stable environment. If all goes well, the absolute earliest it might end up getting pushed out would be sometime tomorrow.

[3rdworld]

Karl Berg, on 23 April 2013 - 11:48 AM, said:

snip

Wow. That is a lot of information. Thanks.

Just curious.

Based on the current network load and such, what do you believe would be the highest number of players you could put into a match?

[Esplodin]

Well, video of minor "jiggles", rather then the much more pronounced out of mech experience others are having:

Edited by Esplodin, 23 April 2013 - 12:19 PM.