10
Spoiler
Matthew Craig, on 23 June 2014 - 12:09 PM, said:
It’s been a while, please pardon the radio silence while we focused on releasing Clans. Starting from that subject, a natural question is: “Public Test went smoother than Launch; What happened launch day?”
During the Public Test for Clans we discovered a serious inefficiency in the script that was injecting clan packages into accounts. Because of the way this script worked, it created a lot of pressure on the database the moment users logged into Public Test. We saw this issue right away and switched the Clan Mechs injection script to use a more optimized method. After this fix, things began to run smoother at our Public Test load; unfortunately, however, there was a deeper issue lurking that wasn't discovered until we moved to production.
When Clans went live on Production, we saw a massive load on our database which didn’t appear in Public Test. To partially alleviate load, first we disabled web access to give the database some breathing room, next our database administrator started looking at what was hammering the database. What he saw was thousands of inventory requests being done per user every time they visited the Mechlab. From there our engineering team did a great job of digging into the inventory system code tirelessly throughout the day to isolate the offending calls.
A bug in the inventory code was causing the UI to download inventory data from the servers much more frequently than required. Until the release of Clans, this bug wasn't causing any problems, because the system was easily able to sustain the additional load created by this bug; however, once Clans were released, this additional load became painfully obvious.
Once the hotfix went live we saw the database load drop by orders of magnitude. As pretty much every operation in the Mechlab involves the inventory the Mechlab, as a whole it became much more responsive and the database has been humming along much happier ever since. Thanks again to the community, it was amazing to see that the majority of the community were very supportive throughout the day and wished us well as we cranked to get everything back up and running. We were also very happy to see that SRMs are now back in play and working normally again.
Now what else has the engineering team been up to? Other than Clans…
Matchmaking
As has already been reported by Russ: The matchmaker code has been revamped and is currently back in testing and making its way towards Production on July 2nd.
UI
With the initial Clans patch out of the way, the UI team has been making some tweaks to UI 2.0 to help usability; some of these have already made their way into test and will be working their way into the July 2nd patch with more to follow in the patches beyond that.
Performance
Some revamping of the HUD was done recently initially focused on making a Clan HUD (Thanks Charla!) and getting 3D Vision working with the HUD. We squeezed a bit more speed out of the HUD but only a minor improvement as we were more focused on the new features. We recognize having gotten more familiar the underlying Scaleform code/assets as part of this work that more significant gains will take additional refactoring and we'll prioritize that as soon as it fits in the schedule. These performance tweaks and 3D Vision support are currently in test and if approved will be delivered in the July 15th patch.
There has also been work done on SLI support that is now complete and again entering testing. It should ensure that users with multiple GPUs can make better use of them to get higher frame rates. As mentioned with the HUD work now complete to support 3D Vision, this gives something fun to throw the extra power of SLI at.
Controller Tweaks
There are some controller tweaks coming to ensure the mouse wheel binds correctly, and to enable absolute inputs for Joystick users along with some other work like allowing you to move while the Battlegrid is up, for example. These are also in test and if approved will be in the July 15th patch.
Stability
We know there are still some stability issues largely surrounding memory usage and a few remaining DX11 bugs. We'll continue to work to track these down and resolve them asap.
Visuals
We're investigating some improvements to the visuals of the game and also still investigating the motion blur bug currently.
Gameplay
As always, there is various gameplay feature work and tuning happening but I'll leave that to Paul to discuss, as you know falling and jump jets have been being looked at recently.
We're very happy that Clans are now playable and it feels great having them stomping around the game. We're glad you're enjoying them as much as we are.
As you're all aware, the effort now shifts to focus on getting the new Matchmaker out and then onwards to Community Warfare.
Thanks again for your support during the Launch of Clans.
See you on the Battlefield!
During the Public Test for Clans we discovered a serious inefficiency in the script that was injecting clan packages into accounts. Because of the way this script worked, it created a lot of pressure on the database the moment users logged into Public Test. We saw this issue right away and switched the Clan Mechs injection script to use a more optimized method. After this fix, things began to run smoother at our Public Test load; unfortunately, however, there was a deeper issue lurking that wasn't discovered until we moved to production.
When Clans went live on Production, we saw a massive load on our database which didn’t appear in Public Test. To partially alleviate load, first we disabled web access to give the database some breathing room, next our database administrator started looking at what was hammering the database. What he saw was thousands of inventory requests being done per user every time they visited the Mechlab. From there our engineering team did a great job of digging into the inventory system code tirelessly throughout the day to isolate the offending calls.
A bug in the inventory code was causing the UI to download inventory data from the servers much more frequently than required. Until the release of Clans, this bug wasn't causing any problems, because the system was easily able to sustain the additional load created by this bug; however, once Clans were released, this additional load became painfully obvious.
Once the hotfix went live we saw the database load drop by orders of magnitude. As pretty much every operation in the Mechlab involves the inventory the Mechlab, as a whole it became much more responsive and the database has been humming along much happier ever since. Thanks again to the community, it was amazing to see that the majority of the community were very supportive throughout the day and wished us well as we cranked to get everything back up and running. We were also very happy to see that SRMs are now back in play and working normally again.
Now what else has the engineering team been up to? Other than Clans…
Matchmaking
As has already been reported by Russ: The matchmaker code has been revamped and is currently back in testing and making its way towards Production on July 2nd.
UI
With the initial Clans patch out of the way, the UI team has been making some tweaks to UI 2.0 to help usability; some of these have already made their way into test and will be working their way into the July 2nd patch with more to follow in the patches beyond that.
Performance
Some revamping of the HUD was done recently initially focused on making a Clan HUD (Thanks Charla!) and getting 3D Vision working with the HUD. We squeezed a bit more speed out of the HUD but only a minor improvement as we were more focused on the new features. We recognize having gotten more familiar the underlying Scaleform code/assets as part of this work that more significant gains will take additional refactoring and we'll prioritize that as soon as it fits in the schedule. These performance tweaks and 3D Vision support are currently in test and if approved will be delivered in the July 15th patch.
There has also been work done on SLI support that is now complete and again entering testing. It should ensure that users with multiple GPUs can make better use of them to get higher frame rates. As mentioned with the HUD work now complete to support 3D Vision, this gives something fun to throw the extra power of SLI at.
Controller Tweaks
There are some controller tweaks coming to ensure the mouse wheel binds correctly, and to enable absolute inputs for Joystick users along with some other work like allowing you to move while the Battlegrid is up, for example. These are also in test and if approved will be in the July 15th patch.
Stability
We know there are still some stability issues largely surrounding memory usage and a few remaining DX11 bugs. We'll continue to work to track these down and resolve them asap.
Visuals
We're investigating some improvements to the visuals of the game and also still investigating the motion blur bug currently.
Gameplay
As always, there is various gameplay feature work and tuning happening but I'll leave that to Paul to discuss, as you know falling and jump jets have been being looked at recently.
We're very happy that Clans are now playable and it feels great having them stomping around the game. We're glad you're enjoying them as much as we are.
As you're all aware, the effort now shifts to focus on getting the new Matchmaker out and then onwards to Community Warfare.
Thanks again for your support during the Launch of Clans.
See you on the Battlefield!
