70 replies to this topic

[Fragger56]

So as we all know hitreg has kind of gone to crap as of late with the optimization tweaks the netcode guys at PGI have implemented.

BUT according to my testing and research of late, its not completely PGI's fault.

Recently one of the awesome devs (Karl) posted that he believes that the issues with HSR we are experiencing currently are due to oversensitivity in the HSR implementation to ping jitter.
After reading this, and remembering the day in closed beta that caustic was the only map available because the air conditioning died at the server farm causing 2/3rds of the servers to overheat. I decided to do some network
snooping in order to try and figure out what the deal was and ran a bunch of traceroutes.

Lo and behold, the 2 routers that sit right before PGI's IP have horrible jitter issues, normally pinging around 50ms to me, yet every 30-90 seconds, i get a response in the 150-200ms range.
After seeing this, I went and had several friends do the same thing from different locations around the world, same issue every time.

So, IMHO the gateway/border routers to whomever the server provider are being run very close to network saturation and the packet shaping equipment is incurring a LOT of jitter to MWO trafficdue to this.
No matter how good PGI makes their netcode, there will be no good way to deal with random 150ms ping jitter that lasts for 1-2 ticks. I'm willing to bet that most every time in the past
that I have noticed shots not registering has coincides with one of these transient pingspikes.

The following two images are screencaps of traceroutes being done over time to the MWO servers, top link is to the traces I did from missouri, the bottom link is to one i had SmokinDave73 do from Australia.
I've also had other clanmates do the same test, with similar results, unusually high amounts of jitter from the last 2 routers before the MWO servers.



So, PGI please for the love of god, move your servers to a REAL data center that doesn't have bandwidth issues...

TLDR:
The pipes to PGI's servers are too small and full of crap, causing random lag that ***** hitreg.
PGI needs better server hosts.

PS:
I intended to post this at the beginning of the week, but with the supposed hotfix in the pipeline, I was going to see if said fix was going to do anything, it didn't, so this post was made.

Edit:
Just wanted to note, the hotfix did help, but it has not resolved the underlying issue, which is connection quality related.
Shots are regging more often, but those with high pings, such as euro and aussie players are still having hitreg issues.

Edited by Fragger56, 21 June 2013 - 01:49 AM.

[SmokinDave73]

I sincerely hope PGI have more netcode/hit reg fixes in the near future. If getting new servers means the game can actually be playable for me again and for everyone else that does not live in America then I hope PGI will consider it.

[AndyHill]

Yeah it's pretty bad at the moment for me. Just had a few games where I registered three or more straight misses against a relatively slow JM and QD builds at less than 300m range. Needless to say I didn't hit the Jenner either and judging by the time it takes to start games the PGI servers are probably not full of people either.

[FREDtheDEAD]

Fragger56, on 21 June 2013 - 01:36 AM, said:

{...}
Shots are regging more often, but those with high pings, such as euro and aussie players are still having hitreg issues.

Tell me about it...

Nice bit of research. Will TCP characteristics always reflect UDP? The game is UDP based. I guess bandwidth saturation is bandwidth saturation.

[Villz]

http://mwomercs.com/...-hit-detection/

Villz, on 21 June 2013 - 05:55 AM, said:

glad to see i'm not the only one experiencing this. And perhaps the problem goes deeper than i thought.

http://mwomercs.com/...roviders-again/

WTB Los Angeles Servers. Hopefully when they eventually release euro servers they will realise that toronto is an awful place for NA / Asia / Oceanic servers. Speaking from an oceanic perspective the 270 ping i get to toronto (middle of nowhere geographically) vs the 170 ping i get to LA (Central Internet Point).

[Corvus Antaka]

interesting find.

[Deathlike]

There is like this one node that I go through in Canada that during the poor netcode time (mainly "dump to mechlab", but most people experienced it as "packet malformed"), it keeps reporting that the node stopped having that connection or something, which leads me to believe that there's something being overloaded.

This is atrocious and needs to be addressed somehow.

Edited by Deathlike, 21 June 2013 - 08:14 AM.

[Fragger56]

Xajorkith, on 21 June 2013 - 04:12 AM, said:

Tell me about it...

Nice bit of research. Will TCP characteristics always reflect UDP? The game is UDP based. I guess bandwidth saturation is bandwidth saturation.

Protocol shouldn't matter, latency is latency.

Also, if the interface is close enough to saturation to incur 150+ms of jitter to shaped TCP traffic, I'd be willing to bet its affecting UDP as well.

Either that or those routers have horrible packet shapers attached to them.

Also, if this is the bandwidth/latency situation now, I can only imagine how bad its going to be when they try to roll out 12v12...

[Deathlike]

Most FPSes AFAIK use UDP for the sake of latency. The last time TCP was used in a MW game IIRC was MW3, but that was coupled with a peer-to-peer network during the dialup era.... and man that lag was bad.

Edited by Deathlike, 21 June 2013 - 08:37 AM.

[Aslena]

This isn't just a problem for people not in North America... I have a great ping "60 to 70 usually" but I still have this problem nonstop every game.

[Neema Teymory]

Fragger56, on 21 June 2013 - 01:36 AM, said:

Recently one of the awesome devs (Karl) posted that he believes that the issues with HSR we are experiencing currently are due to oversensitivity in the HSR implementation to ping jitter

Hey guys, this is true, but its not the whole story. The HSR implementation is complex and as such is affected by many different factors. As for ping jitter specifically, we are planning to make improvements to the system that will reduce or eliminate this sensitivity to ping.

SmokinDave73, on 21 June 2013 - 01:44 AM, said:

I sincerely hope PGI have more netcode/hit reg fixes in the near future. If getting new servers means the game can actually be playable for me again and for everyone else that does not live in America then I hope PGI will consider it.

We are aware of the current problems, and rest asurred we are always working on making things better!

[Fragger56]

Deathlike, on 21 June 2013 - 08:36 AM, said:

Most FPSes AFAIK use UDP for the sake of latency. The last time TCP was used in a MW game IIRC was MW3, but that was coupled with a peer-to-peer network during the dialup era.... and man that lag was bad.

Traceroutes and pings are done in windows via ICMP, so again, latency is latency.
[I think that the linux ping and tracroute tool are UDP]

You can prioritize data, but if there is a queue, you cant really avoid said queue and the data has to wait for some amount of time...

Edit to save from doubleposting:

Neema Teymory, on 21 June 2013 - 09:10 AM, said:

Hey guys, this is true, but its not the whole story. The HSR implementation is complex and as such is affected by many different factors. As for ping jitter specifically, we are planning to make improvements to the system that will reduce or eliminate this sensitivity to ping.



We are aware of the current problems, and rest asurred we are always working on making things better!

I made this post with that in mind, I took a explanation limited in scope, mixed it in with my networking knowledge, made some limited assumptions and started doing some testing that shows an issue.
This thread is meant to point it out to you guys at PGI and I'm grateful that you guys are working on it.

In regards to the "working on it" part specifically:
IMO unless you guys deal with the underlying ping jitter issue, as it IS a localized issue that only appears on the LAST 2 routers before the servers there will never be "perfect" or even "very good" hitreg unless you guys are willing to add ALOT more processing overhead and increase the "buffer" on HSR data.
When 12vs12 is implemented, I expect this overhead increase to be a VERY obvious issue with the way things are now...

With ping jitter of between 600-800 as a fairly common scenario for aussies I'd guess that the HSR system would have to cache about 1500 ms of accurately timestamped client frames. Then it would have to re-organize them into the correct order to account for the jitter, which would only be correctable if the server had a corresponding ping offset for the tick where the jitter happened.
This would likely require the server to compute the RTT of each client frame individually as it is sent and recieved and for the server to store this for HSR computation purposes.

All of that sounds like adding a excessive amount of overhead, both to bandwidth and procecssing power required in order to fix what looks like a simple network bottleneck.

And while Google has proven that enough good software can make ****** hardware work flawlessly, I really don't think PGI should be throwing this much effort into fixing this issue via software when the "hardware" fix should be easier.

Edited by Fragger56, 21 June 2013 - 09:27 AM.

[Lugh]

Xajorkith, on 21 June 2013 - 04:12 AM, said:

Tell me about it...

Nice bit of research. Will TCP characteristics always reflect UDP? The game is UDP based. I guess bandwidth saturation is bandwidth saturation.

yes only UDP will be more adversely affected by the flux. The TCP pipe guarantees packet receipt. UDP sends the packet and does not to confirm it's receipt.

[Tennex]

its a lot better than before from what i can tell (after the hotfix)

[Deathlike]

It could be better... right now my ping is back to 100ms or so at times... I figure this in part has to do with the tourney load, but wtf?

[Bad Andy]

I have a cold beer in my hand right now. Here's to Neema and in general to honest developer to customer communication.

It may not translate directly to making cash dollars, but there are some things more important than that. If you know what they are, you may be at least halfway on your way to making a classic video game.

[malice]

Bumping in hopes that this is researched and acknowledged by PGI.

[FrozenWaltDisney]

malice, on 22 June 2013 - 09:50 AM, said:

Bumping in hopes that this is researched and acknowledged by PGI.

Umm... they responded to this a few posts before. I am sure they are aware of the issue, but if they haven't been able to do anything, reporting that they haven't done anything is counter-productive. I would rather have them working then messaging me on a board.

[shabowie]

Put your servers in Chicago like everyone else in the universe.

[Sudden Reversal]

Reactor - online

Sensors - online

Weapons - online

Lag Shield - online

All systems - SNAFU

Edited by Sudden Reversal, 23 June 2013 - 07:45 PM.