juju2112, on 12 April 2014 - 06:37 AM, said:
Hi Karl!
I have some questions from the perspective of a fellow programmer:
- Beyond just having QA test things, what strategies do you employ to minimize bugs?
- Do you have lots of good unit tests? Is your codebase modular enough for these to be useful?
- When a bug is fixed, is a unit test created to test for it in future builds?
- Are there integration tests?
- Do you have a Continuous Integration environment like Jenkins running?
- How frequently do you need to pull out memory analysis tools like Valgrind, Purify, or GDB?
- Do you have static analysis tools like SonarQube?
Hi juju2112, some excellent questions:
- Beyond just having QA test things, what strategies do you employ to minimize bugs?
We perform regular code reviews before submission. We have an integration pipeline which assists QA by providing them multiple opportunities to test the same code
- Do you have lots of good unit tests? Is your codebase modular enough for these to be useful?
Yup, we have unit tests for some components of our system. These are very low-level tests however, that exercise thread safety on concurrent data-structures, correct behaviour of specific systems, etc.. As to how useful theses tests are, that's difficult to say. They definitely provide more confidence in changes, but we change most of that fundamental code so infrequently that I've never really seen a test fail. In addition unit tests can't really capture interactions between distributed systems, which is the vast majority of our backend architecture; so their utility is definitely limited there. One of our systems engineers has a side project combining which lua tinker and busted to set up high concurrency functional tests. He's actually got this working for the most part. It's a neat system, and I hope we'll start to make much more use of it in future.
- When a bug is fixed, is a unit test created to test for it in future builds?
Sometimes, when it is possible to. Most usually not however, as most bugs require specific edge cases to be set up in a database beforehand, or require some specific interaction of multiple servers to manifest. This is related to the functional testing above.
- Are there integration tests?
We have an integration pipeline as mentioned previously. Moving each stage in the pipeline requires another QA pass and sign-off. We don't really have automated testing set up at this time, although QA has the ability to write / macro test drivers to assist them with 24-player bot testing under various network conditions, or to exercise specific front-end options.
- Do you have a Continuous Integration environment like Jenkins running?
Nope, not at this time.
- How frequently do you need to pull out memory analysis tools like Valgrind, Purify, or GDB?
Every few months I would say. There is no regularly scheduled Valgrind runs, it's on an 'as we remember to do it' basis. That said, it is an amazing tool, and has found some very tricky issues with our code in the past.
- Do you have static analysis tools like SonarQube?
We run CPPCheck against our backend code regularly. We've run it against the engine a couple times, but it's only feasible for us to fix issues discovered in our game logic. The engine itself produces an encyclopedia of output that would take us years of dev time to address.