29 replies to this topic

[GHOST AQC]

Since the last patch, about 1 in 10 games crashes to desktop and always right after the match starts. I use Nvidia cards and have tried the usual restarting, clearing cache and even limited my FPS to 41. Anyone else getting this?

[Karl Berg]

Hey everyone. Just a quick heads up that a major cause of crash to desktop issues was identified from the crash dumps last friday, and a fix has been given to QA for emergency testing and launch. Not sure of when exactly this will end up getting patched out, but this issue is severe enough that will be rushed through with a very high priority.

Detailed reason for the crash if you're curious:

We recently introduced code making use of the MASKMOVDQU SSE instruction to write out only the first 4-bytes of a result from a 16-byte SSE register coming out of a hand-optimized function. Turns out that this instruction pulls in a full 128 bits of data from main memory, does the masked update on CPU, and then writes the full 128 bits back out to main memory regardless of which bytes were actually changed.

I've not made use of this particular instruction before, and the documentation didn't hint that this might be an issue, even though now it seems pretty obvious in hindsight . So if we were unlucky enough that the memory address we were writing back to was aligned right next to another page marked as unwritable, the kernel would generate an access violation.

[KOKO]

Ok, this will fix the CTD issue. If you also can handle the game freeze issue (introduced with the February patch), the game would be stable for me again. I allready sent it to the support (#78619). You can find a LOT of threads in your forums. Best way using Google:

1st: "cryengine command buffer overflow" ... you get this, if you play in window fullscreen, the game hang and closes after confirming the OK button of the error-window.

2nd: "mwo missile freeze": The game just freezes (most of the time on missile hits) Can only be killed in the task manager. This error occurs, if you play in fullscreen mode. I think personally, that 1st and 2nd errors are the same, but you only can see the error message in window-fullscreen.

On both you will find a lot of answers to the thread starters. I hope you also will find the problem soon.

Best regards,

Tim

EDIT: And yes, i tried everything ... cleared the cache very often, different graphic drivers, reinstall, repair tool.

Edited by KOKO, 29 April 2013 - 01:33 PM.

[BlackWidow]

Karl Berg, on 29 April 2013 - 01:08 PM, said:

Hey everyone. Just a quick heads up that a major cause of crash to desktop issues was identified from the crash dumps last friday, and a fix has been given to QA for emergency testing and launch. Not sure of when exactly this will end up getting patched out, but this issue is severe enough that will be rushed through with a very high priority.

Detailed reason for the crash if you're curious:

We recently introduced code making use of the MASKMOVDQU SSE instruction to write out only the first 4-bytes of a result from a 16-byte SSE register coming out of a hand-optimized function. Turns out that this instruction pulls in a full 128 bits of data from main memory, does the masked update on CPU, and then writes the full 128 bits back out to main memory regardless of which bytes were actually changed.

I've not made use of this particular instruction before, and the documentation didn't hint that this might be an issue, even though now it seems pretty obvious in hindsight . So if we were unlucky enough that the memory address we were writing back to was aligned right next to another page marked as unwritable, the kernel would generate an access violation.

You. Are. Awesome.

That is all.

[MajorChunks]

Karl Berg, on 29 April 2013 - 01:08 PM, said:

Hey everyone. Just a quick heads up that a major cause of crash to desktop issues was identified from the crash dumps last friday, and a fix has been given to QA for emergency testing and launch. Not sure of when exactly this will end up getting patched out, but this issue is severe enough that will be rushed through with a very high priority.

Detailed reason for the crash if you're curious:

We recently introduced code making use of the MASKMOVDQU SSE instruction to write out only the first 4-bytes of a result from a 16-byte SSE register coming out of a hand-optimized function. Turns out that this instruction pulls in a full 128 bits of data from main memory, does the masked update on CPU, and then writes the full 128 bits back out to main memory regardless of which bytes were actually changed.

I've not made use of this particular instruction before, and the documentation didn't hint that this might be an issue, even though now it seems pretty obvious in hindsight . So if we were unlucky enough that the memory address we were writing back to was aligned right next to another page marked as unwritable, the kernel would generate an access violation.

I'm really liking all these awesome, highly-technical responses coming out. Please continue. Also, reading the words "hand-optimized" makes me all warm and fuzzy inside.

[Lugh]

Faster please Karl!

[GrizzlyViking]

Just fix it!

[Deathlike]

MajorChunks, on 01 May 2013 - 07:24 AM, said:

I'm really liking all these awesome, highly-technical responses coming out. Please continue. Also, reading the words "hand-optimized" makes me all warm and fuzzy inside.

Hand-optimized is fine.. but you have to VALIDATE to make sure the results are "working as intended". Judging from the technical response, it clearly wasn't.

At least we know they are at least trying to optimize... but they need better tools to make sure stuff isn't broken when stuff is added/changed.

Edited by Deathlike, 01 May 2013 - 06:04 PM.

[SmokinDave73]

Karl Berg, on 29 April 2013 - 01:08 PM, said:

Hey everyone. Just a quick heads up that a major cause of crash to desktop issues was identified from the crash dumps last friday, and a fix has been given to QA for emergency testing and launch. Not sure of when exactly this will end up getting patched out, but this issue is severe enough that will be rushed through with a very high priority.

Detailed reason for the crash if you're curious:

We recently introduced code making use of the MASKMOVDQU SSE instruction to write out only the first 4-bytes of a result from a 16-byte SSE register coming out of a hand-optimized function. Turns out that this instruction pulls in a full 128 bits of data from main memory, does the masked update on CPU, and then writes the full 128 bits back out to main memory regardless of which bytes were actually changed.

I've not made use of this particular instruction before, and the documentation didn't hint that this might be an issue, even though now it seems pretty obvious in hindsight . So if we were unlucky enough that the memory address we were writing back to was aligned right next to another page marked as unwritable, the kernel would generate an access violation.

I have not experienced a crash to desktop since closed beta, now I carash to desktop about 1 in 5 games how does this coding get past a internal alpha server??? Plus is this not a high priority bug yet im pretty sure 90% of the player base is getting crash to desktop now.....

[MajorChunks]

Deathlike, on 01 May 2013 - 06:04 PM, said:

At least we know they are at least trying to optimize... but they need better tools to make sure stuff isn't broken when stuff is added/changed.

True, but I've debugged low-level C and NASM assembly memory problems before, and I almost threw myself off of a bridge because of it. The fact that these guys can deal with it at all, and have the stones to admit when they messed up, earns them huge points in my book. You're right, it doesn't stop them from needing better regression testing, but we already knew that

Edited by MajorChunks, 02 May 2013 - 06:50 AM.

[DragonsFire]

SmokinDave73, on 02 May 2013 - 06:48 AM, said:

I have not experienced a crash to desktop since closed beta, now I carash to desktop about 1 in 5 games how does this coding get past a internal alpha server??? Plus is this not a high priority bug yet im pretty sure 90% of the player base is getting crash to desktop now.....

Unfortunately internal test servers are not the same thing as a real world deployment and as a result things can and do get missed. It's the nature of the beast, but it's also why we're here. As for the actual CTD, I too haven't had much of it since CB, but after the most recent patch, I had 2 happen pretty close to one another. Did a full uninstall and reinstall of the client and all is right with the world again, at least for me.

[Karl Berg]

The function was in fact performing perfectly fine. For a given set of inputs, the output was exactly what was expected; and we did run unit testing to confirm this. Unfortunately the chance of crashing was extremely random, since it mattered where in memory the function was told to store its results, and even if the return address crossed a page boundary, what the access permissions of that next page were.

Of course, none of this excuses us from missing this issue during testing; and for that we definitely apologize. The hotfix for this has made its way all the way to stage, and we are planning to patch this on production tomorrow morning.

[SmokinDave73]

Karl Berg, on 02 May 2013 - 10:00 AM, said:

The function was in fact performing perfectly fine. For a given set of inputs, the output was exactly what was expected; and we did run unit testing to confirm this. Unfortunately the chance of crashing was extremely random, since it mattered where in memory the function was told to store its results, and even if the return address crossed a page boundary, what the access permissions of that next page were.

Of course, none of this excuses us from missing this issue during testing; and for that we definitely apologize. The hotfix for this has made its way all the way to stage, and we are planning to patch this on production tomorrow morning.

Thanks for the info, and I do understand that errors from time to time get through testing its just the CTD is by far the most frustrating bug of them all. Hopefully this hotfix helps the situation.

[Jack Lowe]

Excellent Karl thank you very much for the update, and particularly the explanation. Very helpful and for such an annoying bug it actually helps to know what the problem was. The fix may in fact affect some of the other bugs related to CTD. I'll certainly put in a few more games on my days off next week and see how it goes. If anything new pops up or some known problems seem less frequent I'll put it up here someplace.

[Krzysztof z Bagien]

[MajorChunks]

Karl Berg, on 02 May 2013 - 10:00 AM, said:

The function was in fact performing perfectly fine. For a given set of inputs, the output was exactly what was expected; and we did run unit testing to confirm this. Unfortunately the chance of crashing was extremely random, since it mattered where in memory the function was told to store its results, and even if the return address crossed a page boundary, what the access permissions of that next page were.

Of course, none of this excuses us from missing this issue during testing; and for that we definitely apologize. The hotfix for this has made its way all the way to stage, and we are planning to patch this on production tomorrow morning.

You guys rock. Don't let anyone tell you otherwise.

[Naotaka]

Karl Berg, on 29 April 2013 - 01:08 PM, said:

it seems pretty obvious in hindsight

Hindsight is 20/20 my friend. Chalk it up as useful experience gained.

Keep rocking.

[Karl Berg]

The hotfix is up now, so this specific crash to desktop should be fixed. As always, it's a big help to us if you file a report and submit a crash dump when you do encounter a crash.

Thanks!

[Lugh]

Woot so I can patch and play and hopefully not see any more CTDs..

[Karl Berg]

Ahh, well I don't think I can reasonably claim that there are no more crash bugs with a game of this size and complexity. Especially as we dig deeper and deeper into these outstanding scaleform/CryEngine/HUD issues. But this specific crash issue which was introduced a few weeks ago is now fixed; and it was a pretty bad one.