292 replies to this topic

[Void Angel]

Shumabot, on 15 May 2013 - 12:46 PM, said:

Could you tell us what the priority for this issue looked like over the past few months? Was it a background issue until tools were developed to deal with it? Did players start encountering the issues more than previously which forced fixes to be expedited? Did staff free up from other development projects and were moved into this nagging issue as able?

I... can't help but notice that any answer to these questions could be used as grounds for armchair developing and complaints.

[Shumabot]

Void Angel, on 15 May 2013 - 03:22 PM, said:

I... can't help but notice that any answer to these questions could be used as grounds for armchair developing and complaints.

Possible, they seemingly avoided answering any of them in the initial superpost which mostly stressed how hard the fix was. I think they did a great job in tracking down a very obscure conflict, but the fact remains that it took them many months to fix it. A confirmation of any of those outlined scenarios isn't really damning, it's a game under heavy system development and they're all valid reasons for why something like this could have occurred along the timescale that it did (this isn't fractalgate or SRM splash feature). Two of them (freed up talent and developed testing resources) actually imply good things about their current development practices and show an evolving behind the scenes effort to tackle day to day issues. Silence and simply brushing the time frame under the rug makes it seem like they didn't actually prioritize fixing this issue until recently.

Edited by Shumabot, 15 May 2013 - 03:28 PM.

[Void Angel]

Just confirming that your oh-so-innocently phrased questions were the passive-agressive criticism and armchair quarterbacking they appeared to be. Good day.

[Shumabot]

Void Angel, on 15 May 2013 - 03:55 PM, said:

Just confirming that your oh-so-innocently phrased questions were the passive-agressive criticism and armchair quarterbacking they appeared to be. Good day.

Ahh, you're a sycophantic fanboi who lacks the ability to reason (or even read considering two of the options were praiseworthy and I made sure to explain to you why). You don't have a gold tag, you must have been pressganged by Bishop Steiner and then struck in the head. It's a big step for PGI to actually relate whats going on behind the scenes, wishing for them to give the whole picture and express their intentions as developers isn't passive aggressive criticism.

Edited by Shumabot, 15 May 2013 - 04:06 PM.

[WVAnonymous]

Shumabot, on 15 May 2013 - 04:04 PM, said:

Ahh, you're a sycophantic fanboi who lacks the ability to reason (or even read considering two of the options were praiseworthy and I made sure to explain to you why). You don't have a gold tag, you must have been pressganged by Bishop Steiner and then struck in the head. It's a big step for PGI to actually relate whats going on behind the scenes, wishing for them to give the whole picture and express their intentions as developers isn't passive aggressive criticism.

He took a refund on his founders, IIRC...

Unless that was M A L I C E?

Edited by WVAnonymous, 15 May 2013 - 04:16 PM.

[Thedrelle]

[Shumabot]

WVAnonymous, on 15 May 2013 - 04:15 PM, said:

He took a refund on his founders, IIRC...

Unless that was M A L I C E?

I am getting master class rused.

[Biglead]

I just read your briefing.

Don't know what the f*%k it said, but thank you guys for all your hard work.

[Narcissistic Martyr]

As far as I can tell, Karl is a witch doctor and he performed ritual magic in a strange tounge to drive out the demons the great code creators put there to test us.

Good job magic men.

[3Xtr3m3]

WTG Karl!!

I understood as much as I could and took the rest on faith.
If I had one question it would be ...

Just how long did it take to write that explanation?

[Deathlike]

Shumabot, on 15 May 2013 - 12:46 PM, said:

Could you tell us what the priority for this issue looked like over the past few months? Was it a background issue until tools were developed to deal with it? Did players start encountering the issues more than previously which forced fixes to be expedited? Did staff free up from other development projects and were moved into this nagging issue as able?

Without having to watch for an answer to even defend himself, it should be apparent that the issue got worse (like the whining), thus making it a lot more imperative to get it fixed. You're essentially asking a rhetorical question for the most part.

It doesn't sound like more tools were created, but moreso analysis of the code that was being debugged. Tracing code behavior can be slow so these things tend to go step by step, line by line. This of course assumes that the code provided actually adhere to the limits/restrictions that is required for them to work. From the description, that was the process, including putting together a special build internally that would be painfully slow to run, but that isn't for us to experience.

Quote

We first spent a considerable amount of time attempting to reproduce the issue with a debug build and breakpoint a failure case, in the hopes that the error was obvious, but also to get an idea where the issue truly resided so we knew where to focus our efforts. Eventually we did manage to reproduce one of these issues with a debug build, and what this effectively confirmed for me was that there was almost certainly an issue somewhere in CryEngine itself. Starting from here, it became important to understand as much about the interactions between CryEngine and Scaleform as possible.

What followed was about two days of line by line debugging and auditing code in order to figure out what the intended behaviour of these systems were. I’ve got several pages of notes from this process, outlining general flows and several class structures. Once I had a good idea of how this system was intended to work, I spent some time building up a set of validation and verification routines to detect when these subsystems entered an invalid state. I was specifically interested in a convoluted set of lookup structures which mapped string handles to Scaleform object instances. The end result was an extremely slow build, but which would crash and generate stack call data whenever it detected that the CryEngine scaleform layer had entered an invalid state.

Edited by Deathlike, 15 May 2013 - 06:21 PM.

[Devil Fox]

3Xtr3m3, on 15 May 2013 - 05:57 PM, said:

WTG Karl!!

I understood as much as I could and took the rest on faith.
If I had one question it would be ...

Just how long did it take to write that explanation?

Probably not long... might find he just cobbled together the ongoing bug notes they were keeping on its progress into some coherent state!

[Void Angel]

Shumabot, on 15 May 2013 - 03:27 PM, said:

Possible, they seemingly avoided answering any of them in the initial superpost which mostly stressed how hard the fix was. I think they did a great job in tracking down a very obscure conflict, but the fact remains that it took them many months to fix it. A confirmation of any of those outlined scenarios isn't really damning, it's a game under heavy system development and they're all valid reasons for why something like this could have occurred along the timescale that it did (this isn't fractalgate or SRM splash feature). Two of them (freed up talent and developed testing resources) actually imply good things about their current development practices and show an evolving behind the scenes effort to tackle day to day issues. Silence and simply brushing the time frame under the rug makes it seem like they didn't actually prioritize fixing this issue until recently.

Void Angel, on 15 May 2013 - 03:55 PM, said:

Just confirming that your oh-so-innocently phrased questions were the passive-agressive criticism and armchair quarterbacking they appeared to be. Good day.

Shumabot, on 15 May 2013 - 04:04 PM, said:

Ahh, you're a sycophantic fanboi who lacks the ability to reason (or even read considering two of the options were praiseworthy and I made sure to explain to you why). You don't have a gold tag, you must have been pressganged by Bishop Steiner and then struck in the head. It's a big step for PGI to actually relate whats going on behind the scenes, wishing for them to give the whole picture and express their intentions as developers isn't passive aggressive criticism.

Asking "innocent" questions in order to find fault is, however. But that's par for the course with the Professionally Dissatisfied, as is your resort to a load of ad hominem, straw man horsecarp. Since your intent wasn't to actually understand anything, but rather to gain ammunition to fuel something you'd already decided to complain about (the time it took to fix,) my initial estimate of your character stands. Pity; sometimes I'd rather be wrong. /ignore

[Hammerfinn]

Shumabot, on 15 May 2013 - 04:04 PM, said:

Ahh, you're a sycophantic fanboi who lacks the ability to reason (or even read considering two of the options were praiseworthy and I made sure to explain to you why). You don't have a gold tag, you must have been pressganged by Bishop Steiner and then struck in the head. It's a big step for PGI to actually relate whats going on behind the scenes, wishing for them to give the whole picture and express their intentions as developers isn't passive aggressive criticism.

Did you actually READ the blow-by-blow? You have no understanding of how long this stuff takes if you REALLY think they could have done it faster. You can't pop code into the microwave, hit a button, and make it right.

[Name140704]

Hats black and red off. Excellent read.

[NemesisXIII]

Thanks for the really interesting read, glad you finally found it. Even more elated that it turned out not to be in your code proving that you guys really are trying to make the game better for the rest of us.

[Thorn Hallis]

I don't even understand half of what Karl is talking about, but great job anyway! Thanks for the writeup.

And another thanks to the wannabe-cynics - you still amuse me.

[Ralgas]

Shumabot, on 15 May 2013 - 03:27 PM, said:

Possible, they seemingly avoided answering any of them in the initial superpost which mostly stressed how hard the fix was. I think they did a great job in tracking down a very obscure conflict, but the fact remains that it took them many months to fix it. A confirmation of any of those outlined scenarios isn't really damning, it's a game under heavy system development and they're all valid reasons for why something like this could have occurred along the timescale that it did (this isn't fractalgate or SRM splash feature). Two of them (freed up talent and developed testing resources) actually imply good things about their current development practices and show an evolving behind the scenes effort to tackle day to day issues. Silence and simply brushing the time frame under the rug makes it seem like they didn't actually prioritize fixing this issue until recently.

It's a grey area loaded question though and any answer given will only lead to more criticism, so they are probably right in not answering either way.

Edited by Ralgas, 15 May 2013 - 11:07 PM.

[MikeFreeman]

Being a developer on my own I really can imagine how frustating and time consuming this investigation was, especially when you come to the point that a third party is causing the issue. But even more I appreciate the detailed post about the whole bugfixing story, excellent!

[ShadowDarter]

Can you please include some more of these types of answers into the general pool of knowledge as to inform people what you guys do behind the scenes, and please thank the Q&A staff for the great work.