75 replies to this topic

[Anais Opal]

This is by no means 100% official as obviously I'm not PGI staff and those who know me on these forums know I'm probably one of PGI's harshest critics, but I have been authorised by a PGI staff member to pass on some information about the problems with people being unable to patch their clients and get down to serious Mech stomping.

First off, let me explain how the patches and updates are delivered.

PGI uses a third party Content Delivery Network or CDN called Akamai, and it happens to be probably the largest one available, a good majority of software and game developers use it, particularly Microsoft ( you know, that annoying downloader you have to install if you are using the Microsoft Developer Network? That's Akamai )

Now, to ensure high performance content delivery, Akamai spread their content across multiple servers in multiple countries, when the patcher makes a content check the patcher URL resolves to the NEAREST akamai content delivery server. If you do an NSLOOKUP you'll see the that first entry is,in PGI's case, a1982.d.akamai.net, then the IP's associated with that ID that are NEAREST to the requesting client. Note, a1982 is the ID that is used to identify content specific for PGI and then with a bit of clever DNS jiggery pokery Akamai selects the physical boxes where the patches are held. It is different for each region, US North, South, Europe, Asia, Australia. These boxes may be in your country or may be a few thousand kilometers away but they are the closest.

In the case of yesterdays patch, everything went fine at PGI's end, servers updated fine and came online and all appeared well. What WASN'T expected was that Akamai themselves internally made some change to the content network which had a wild backlash onto the MWO Community.

This, unfortunately, was unforeseen and COMPLETELY out of PGI's control and currently remains out of PGI's control.

They want to assure us all that they are trying very hard to get the situation resolved and that it is top priority at this time.

PGI want to reiterate that they are very sorry for the trouble (and bad feeling) this has caused and as stated by Kyle, they will reimburse lost ACTIVE Premium time on a case by case basis so please submit a support ticket to support@mwomercs.com when you have determined how much Premium Time you have lost as a result of the issue.

In the meantime, to those who are unaffected.

HAPPY MECH STOMPING!

oxoxoxox

Edited by GlycerineOxide, 16 July 2014 - 11:15 AM.

[Hades Trooper]

well thats just friggin crap arse typical.

[Lokesh]

They pay Akamai for the service, they certainly DO have control as they are the customers in that relationship and are apparently not getting good service. I haven't ever taken issue with PGI before, but the attitudes that I saw communicated by customer support and the devs here on the forum was deplorable. Saying, "go blame your ISP," and being rather rude about it is completely unprofessional. I don't particularly have a recourse, but if this had been pre-clan release I would have cancelled my order.

[Datamatrix]

Hi all
I have just done a fresh install on 2 pc's which i have used for this game and worked fine. 1 has fresh install and is working correctly - the second fresh install which is the one i use the most state that there is maintenance still going on so no issue with isp

Edited by Datamatrix, 16 July 2014 - 11:57 AM.

[mad kat]

Cheers thanks for the update it's good to know that PGI have the transparency to say this themselves....Not.

Neither my computer's clients are working ho hum.

Appreciate the update OP.

Edited by mad kat, 16 July 2014 - 12:06 PM.

[Cysero]

Seems like rather a lot of people have tried a fresh install (even with Staff telling them not to) and it generally changes nothing. Do hope this gets resolved soon, my Premium time in burning and I've already chalked up today's 2X to a loss.

Datamatrix, on 16 July 2014 - 11:56 AM, said:

Hi all
I have just done a fresh install on 2 pc's which i have used for this game and worked fine. 1 has fresh install and is working correctly - the second fresh install which is the one i use the most state that there is maintenance still going on so no issue with isp

[Kibble]

So is this what is causing the huge amounts of lag spikes?

[Ninthshadow]

Off I go to do something else then,

Well, it's reassuring to be somewhat "in the loop." Thanks for sharing.

The other thread was pretty useful too, as a 'play by play' of the incident goes.

[Anais Opal]

Kibble, on 16 July 2014 - 12:08 PM, said:

So is this what is causing the huge amounts of lag spikes?

Sorry, I have no information on that problem

[Anais Opal]

Admin, pls pin this.

[Jakob Knight]

GlycerineOxide, on 16 July 2014 - 10:52 AM, said:

This, unfortunately, was unforeseen and COMPLETELY out of PGI's control and currently remains out of PGI's control.

While I respect the time and effort that you put into helping with this issue (and your knowledge of the subject matter was sorely needed by the community on this one), I can't help but think the above statement is not entirely correct. It was PGI's decision to use the third-party service they did, so the responsibility for that decision and any consequences following it are still PGI's. While they may have ceded control of their update service infrastructure to Akamai, it was still their responsibility to oversee the process in its entirety. Outsourcing is never an excuse for failure to discharge one's responsibilities.

Added to that was the conduct of PGI when the problem was reported. Instead of looking at what they had done in the patch process that might have caused the problem, they insisted people switch to Google DNS. Then we get statements about how people 'may enjoy some of the benefits as well' of using Google, then claiming it's the fault of the users for using other providers (Kyle's post about "it's not our problem, go look to your ISP as the culprit" is rather blatant about that). All of this gave the real impression PGI was attempting to push people out of their current ISP and to Google. This was very unprofessional behavior, especially given the fact that the problem was only produced when PGI executed their patch. Attempting to deflect blame by claiming it was nothing they did when it was clearly their actions that caused it only added to the problem.

In summary, I understand they are still working on this problem, but I cannot accept the subject is completely out of their control and was unforeseen as it is PGI's responsibility to both maintain control of their product and foresee just this kind of problem. Also, the conduct of PGI in this matter was less than professional and promoted impressions of strong-arm tactics as well as a sense they were trying to absolve themselves of their responsibilities, even if neither of these were the factual case.

Edited by Jakob Knight, 16 July 2014 - 01:55 PM.

[Anais Opal]

Jakob Knight, on 16 July 2014 - 01:53 PM, said:

While I respect the time and effort that you put into helping with this issue (and your knowledge of the subject matter was sorely needed by the community on this one), I can't help but think the above statement is not entirely correct. It was PGI's decision to use the third-party service they did, so the responsibility for that decision and any consequences following it are still PGI's. While they may have ceded control of their update service infrastructure to Akamai, it was still their responsibility to oversee the process in its entirety. Outsourcing is never an excuse for failure to discharge one's responsibilities.

Added to that was the conduct of PGI when the problem was reported. Instead of looking at what they had done in the patch process that might have caused the problem, they insisted people switch to Google DNS. Then we get statements about how people 'may enjoy some of the benefits as well' of using Google, then claiming it's the fault of the users for using other providers (Kyle's post about "it's not our problem, go look to your ISP as the culprit" is rather blatant about that). All of this gave the real impression PGI was attempting to push people out of their current ISP and to Google. This was very unprofessional behavior, especially given the fact that the problem was only produced when PGI executed their patch. Attempting to deflect blame by claiming it was nothing they did when it was clearly their actions that caused it only added to the problem.

In summary, I understand they are still working on this problem, but I cannot accept the subject is completely out of their control and was unforeseen as it is PGI's responsibility to both maintain control of their product and foresee just this kind of problem. Also, the conduct of PGI in this matter was less than professional and promoted impressions of strong-arm tactics as well as a sense they were trying to absolve themselves of their responsibilities, even if neither of these were the factual case.

I have to admit, for myself the jury is still out on this.

If I was staff, I would have access to more information which I would gladly give out (probably get instantly fired because I believe in 100% transparency unless it's NDA I do follow rules..to a point.i.e. they make sense)

I do agree, IGP support can be less than exemplary (A member of the DevOps team contacted me directly so I didn't have to deal with IGP)

However, I can see the logic here, I can see how using Google DNS would help some people, particularly in the US as it means there is less hops to perform to find the required host. Cross Pacific/Atlantic I can understand why it works in only SOME cases, those cases are dependant on the hops required between your client to the server.

In the case of the CDN, It is my belief that the patch has not populated across all of the content network so only some servers have full patch data and some don't so the client thinks they are offline. In that case the problem is at Akamai as the problem is internal to their network segment.

I'm in the unfortunate position of being able to see and understand the Communities and PGI's assessments of the situation.

Edited by GlycerineOxide, 16 July 2014 - 02:12 PM.

[Cid Slayer]

I am unable to log in and changing the DNS to Google and other methods I've found on here haven't worked to resolve it.

[Anais Opal]

Cid Slayer, on 16 July 2014 - 02:25 PM, said:

I am unable to log in and changing the DNS to Google and other methods I've found on here haven't worked to resolve it.

No, for many there is still problems, myself included.

Maybe PGI will tell me when they think this is going to be solved, maybe not. I'd rather they Command Chair posted the resolution.

[Krist Smith]

Ok, so here's the final report. The problem on our side was expecting standardized error codes to be used. I'll leave ISPs out of this post, except to say that some don't like to return valid codes.

At any rate, the maintenance mode detection was failing due to this - we were expecting a 404 error (page not found) when the file was not there, and we were instead getting a 200 (OK) response. I'm in the process of updating the patcher right now to fix this problem and should be done within half an hour to an hour (we don't want to rush a fix that breaks everything else). I just wanted to update the status immediately, now that we have something to actually report.

The repair tool will to the trick in fixing things once I'm done. I'll post a link once the fix is in (unless someone else wants to while I work on the fix).

[jackal40]

Krist Smith, on 16 July 2014 - 02:52 PM, said:

Ok, so here's the final report. The problem on our side was expecting standardized error codes to be used. I'll leave ISPs out of this post, except to say that some don't like to return valid codes.

At any rate, the maintenance mode detection was failing due to this - we were expecting a 404 error (page not found) when the file was not there, and we were instead getting a 200 (OK) response. I'm in the process of updating the patcher right now to fix this problem and should be done within half an hour to an hour (we don't want to rush a fix that breaks everything else). I just wanted to update the status immediately, now that we have something to actually report.

The repair tool will to the trick in fixing things once I'm done. I'll post a link once the fix is in (unless someone else wants to while I work on the fix).

So was Kyle wrong to blame our ISPs for this issue? If so, I expect a public apology from both Kyle and Russ. I certainly want to hear more about this issue and how PGI intends to prevent similar occurrences.

[Krist Smith]

jackal40, on 16 July 2014 - 02:56 PM, said:

So was Kyle wrong to blame our ISPs for this issue? If so, I expect a public apology from both Kyle and Russ. I certainly want to hear more about this issue and how PGI intends to prevent similar occurrences.

You misunderstand, I simply wanted to leave outside blame out of my post. I didn't say their breaking standard error codes wasn't the root of the problem.

[Anais Opal]

So,

By my estimation, in one of my posts I figured that patch data was only partially populated.

So from Krist is saying, though the patcher had a coding problem which couldn't detect a missing file, the issue still lay in the CDN server configuration?

EDIT:

Krist, I have a suggestion for you.

For build detection, have a buildData.xml file on the login server, the patcher checks itself against this file. If the build matches, go ahead and load the front end, if it fails go ahead and attempt to download a patch from where ever. Having a central build XML would also allow you to pre-package updates so if updates by the normal method fail a player can go to mwomercs.com/patches and download a patch direct and apply that to the installed client.
Re-launching then checks against the build on the login server.
That gives you redundancy.

Edited by GlycerineOxide, 16 July 2014 - 03:17 PM.

[jackal40]

Krist Smith, on 16 July 2014 - 02:58 PM, said:

You misunderstand, I simply wanted to leave outside blame out of my post. I didn't say their breaking standard error codes wasn't the root of the problem.

Well, this needs to be spelled out - where does the blame for the problem lie and what will be done to prevent it in the future.

On a somewhat side note, I understand that many game developers and other publishers use CDNs, yet in many years of game playing as well as almost ten in Network/Server Administration I've never heard of this issue before. This is why I'd like more info.

[Krist Smith]

GlycerineOxide, on 16 July 2014 - 03:00 PM, said:

So,

By my estimation, in one of my posts I figured that patch data was only partially populated.

So from Krist is saying, though the patcher had a coding problem which couldn't detect a missing file, the issue still lay in the CDN server configuration?

Um, not really the CDN so much. Standards dictate that if a page doesn't exist, it returns the 404 error code. This is not followed by some ISPs. These will return a page that says "we couldn't find the page" but the response code, which should be 404, is instead 200 (which means page found, no problems anywhere). Again, I'm not a fan of the blame game and will accept my fair share on this.