292 replies to this topic

[Muzakman]

Great retro on that issue! It's no wonder that the proximate cause wasn't found until recently.

I'm very impressed with the approach and thought process; when I interview devs, I'm always hoping to elicit signs of that kind of thought process from them. The first impulse wasn't to "blame the compiler" for the issue. Even after evidence began to mount, there was appropriate skepticism towards blaming the lower-level abstractions in the codebase, since hey - what's more likely: a defect presents itself in the midst of a codebase not experiencing a lot (if any) churn, or a defect presents itself in the midst of code having a repos history resembling the British Royal Family Tree?

So the problem wasn't (strictly speaking) a corrupted pointer or v-table then, eh? I had figured it would be something along those lines when I would see wrong data in the HUD, like 'undefined'.

Reading your post reminded how ScaleForm is essentially a Flash (render?) host IIRC.

Since SF is (renders, executes, whatnot) Flash, doesn't that mean that the HUD scripts, are all ActionScript, which is essentially JavaScript? Is SF and Flash just acting then as an interpreter for ECMAScript and CryEngine, or is that layer actually responsible for rendering itself based upon data exposed and retrieved from the engine?

It must've been *incredibly* frustrating when it started to become obvious where the issue was. Even the fanciest tools at your disposal would have been/are powerless to detect that kind of issue!

You'd essentially need to have memorized parts some or all of the actual memory addresses assigned at compile-time! Given ISE and such, those addresses have the potential to be different depending on a number of factors not in your control.

Anyways, I'm glad that your efforts have been successful. It's a great feeling when you can make misbehaving code your ******!

Cheers to the team for producing a terrific game and all the hard work both behind and ahead!

[Dmoose]

Wow great story. We rarely get to see into other programmers woes. There nothing like the feeling of squashing such a persistent and hard to track down bug.

Does make me glad I don't code in C++.

[acemarke]

As a fellow programmer, you have my sympathy and my deepest respect for that debugging recap. That is some incredibly deep spelunking into some very dark corners of CryEngine and C++. Fantastic work.

While you certainly don't have to go into this depth all the time, bug fix recaps are always appreciated, both as a gamer and as a developer. Looking forward to more of these in the future!

[Asmudius Heng]

That is an amazing amount of detail - I would have been happy with "Coding is hard" and thats that.

I wish gameplay and balance issues were given the same level of detail to the community ...

[Kiiyor]

Zynk, on 14 May 2013 - 02:04 PM, said:

Next time you give an explanation for a bug please use the K.I.S.S. method and give me the CliffsNotes in the first paragraph with the engineers version after.

Hell no! I like this sort of thing to read like an epic. Give me all the gory, frustrating details!

[SvennK]

Karl Berg's post on the HUD bug is THE best post on this forum to date. Well done, and thank you for a serious, and extremely informative post.

[ChargerIIC]

"It looks like we might have a trace on the HUD Bug. Looks like a problem with the Cryengine Scaleform library and the STL data structures."

"I see..here's what we are going to do. Burn the SBIs, the PBIS, and any defect reports. Destroy the support tickets and fire any member of client services who's mentioned the issue. Erase your workstation and hire assassins to kill anyone who submitted a support ticket. Then deny everything. "

"That sounds like a lof of work - wouldn't it be easier to fix the bug?"

"You mentioned STL?"

"Damnit you're right. The kill orders will be out by Friday"

[Hawkwings]

Hot damn. THIS is how you write a Command Chair post. Thanks for informing us in excruciating detail about this whole endeavor. And keep up the good work!

[Fenris Krinkovich]

I love this kind of stuff. I love seeing what's going on behind the scenes.

Please include Karl's post as part of your auto-reply for bug reports.

I wish we could forcefeed this to the ragers.

[slide]

Thank you for the explanation.

My eyes glazed over and my brain melted about half way through, but I would rather that than a "we found it, fixed" type of post.

A thousand more thankyou's.

[Hawkeye12]

HUD was a very rare bug for me, maybe 2 or 3 times out of 100 matches..I just never got the HUD bug!

Nice Work!

[Arcturious]

Just like to add my appreciation of the work behind the scenes. It's posts and detailed info like this that make me happy I'm a founder.

Now I'd love to see a similar one that goes over the work required to rewrite the missile code and show why strange behaviour was seen when splash damage was reduced. That would be another one that shows the community why these sorts if things can't be fixed overnight, and that you haven't been ignoring the issues but working hard behind the scenes.

Kudos!

[Chemie]

My take aways:

* "It's complicated" (hence why it took so long)
* It's not our fault (blame cryengine)

However, as others have pointed out, many cryengine games have HUD without issue. Also, for me, this issue started in january. Never saw it in CB or early OB. That tells me you did SOMETHING...not saying it also wasnt the cryegine, but some interaction with how you did things.

As stated before, I just hope ALL HUD issue, like losing friend/foe is fixed too.

Thanks

[Draco Harkins]

Karl Berg, on 14 May 2013 - 11:32 AM, said:

Nope, we're using an official release. Turns out some of this code has even been posted online, so if you're curious, here is the global string table and CCryName class itself:

http://www.gamedev.s..._8h_source.html

This is only a piece of the puzzle unfortunately. The rest lies in their usage of STLPort and the non-public CryScaleform library which combined these pieces together.

Give this man and his team a raise. That is all.

[Codex Clavdex]

As another fellow programmer It was a very good read and a amazing level of debugging/QA work performed. I know that bug elimination can be some of the hardest work in the development life cycle and for you to take the time to let us know the extent to which you all are going to squash these pesky little buggers means a lot. Keep up the good work and entertaining reads!!

[Nothing Whatsoever]

Sweet, thanks for explaining the process for hunting down this bug. Learning these kinds of details is really cool IMHO.



Edit: I gotta figure out why spell checker ain't checkin' in my browser.

Edited by Praetor Shepard, 14 May 2013 - 05:04 PM.

[Kaemon]

always a bug hunt...nice work team!

[jay35]

New bug since the hotfix patch that I've never had before: In the middle of the fourth match I played, the game screen goes black, game is still going on in the background. I cannot see anything but I can still hear the battle continuing. If I hit Escape, I can see the game cursor on the screen, but cannot see anything to click on it. Only way to get out of it is to Alt+F4.

This is playing the game in Windowed mode which is how I've always played it.

Edited by jay35, 14 May 2013 - 05:27 PM.

[Ceesa]

Even though I didn't understand the details, I was able to follow the major plot, and I want to give a giant thank you to everyone who worked on that.

Please continue to post these sorts of things when you find and resolve major bugs. There's really no such thing as being too transparent with the community.

[Stargoat]

Thank you for regaling us with this saga of epic bug-hunting, and congratulations on finally squashing one of the most infuriating bugs in the game.