326 replies to this topic

[martian]

Thunderbuddy, on 03 December 2023 - 10:05 AM, said:

As a brand new player with 1 mech, it is very offputting to have my mech locked and not be able to locate matches. This game is way too old to be having open beta level issues.

Read this explanation, please: Mwo Status Update And Known Issues - Update 2 - Dec 1St

The situation should be resolved in ten days.

[Mechwarrior2342356]

Ten days with the ability to occasionally get matches until then isn't terrible though I'm kind of cheesed off that I can't play any of my 4 Centurions until then (AH, D, YLW, AL)

[CrustyMech]

Besh, on 02 December 2023 - 11:15 PM, said:

But why does it load the entire Database at match/lobbycreation ? Why does it not just pull whats needed for the specific 24 'Mechs/loadouts/BoltOns etc. neeed for that specific Match ?

It could be a limitation on the engine.... it might be the way the game was originally coded and the complexity of modifying the code is not feasible.... or something else . I spent 40 years in IT including working with huge/complex database systems for our local health care system. There was NEVER a quick fix...

Edited by CrustyMech, 03 December 2023 - 10:35 AM.

[Maxx31]

What about any Premium Time that was burned due to the issues? I would say I agree there is not a 100% allocation (games that do run, sleep, eating, etc).

[Strelok7]

Besh, on 02 December 2023 - 11:15 PM, said:

But why does it load the entire Database at match/lobbycreation ? Why does it not just pull whats needed for the specific 24 'Mechs/loadouts/BoltOns etc. neeed for that specific Match ?

I feel like a "very loaded answer" today - here you go.
Binary logic vs human logic (*although humans do use it and fall for it in political populism...for example).
I think there is you answer - pull > pull from what > database > where is the database > lets load it > operation success
why is it not always loaded? > server resources should not be always loaded (waste) RAM vs HDD
Human brain example - we don't need to walk around all day, thinking about making a burger, when we should be fixing our car. That is ADD. Usually caused by Stress and Anxiety. *Rest is the solution. Straight functioning brain keeps priorities strait. Just in time information.

Matt Newman, on 03 December 2023 - 12:33 AM, said:

Also I am glad you like my Kitchen Metaphor I will say that yes as humans we don't have to load every ingredient when we are making a specific dish! BUT think about how many dishes you know how to make! Or how you know what spices and flavors to add to get a specific flavor its almost like you DO have a Database in there right?

Brilliant metaphor I was conveying your metaphor to someone who is not too interested in this stuff this morning - they loved it too!

[martian]

Maxx31, on 03 December 2023 - 10:39 AM, said:

What about any Premium Time that was burned due to the issues? I would say I agree there is not a 100% allocation (games that do run, sleep, eating, etc).

Read this, please:

Piranha Games Inc. said:

Once the fix is deployed, we plan to host an event with compensation as a gesture of appreciation.

The compensation event and all the Holiday events will not be posted or started until the patch is live, and we can confirm on the live environment that everything is back to normal.

[martian]

the check engine light, on 03 December 2023 - 10:29 AM, said:

Ten days with the ability to occasionally get matches until then isn't terrible though I'm kind of cheesed off that I can't play any of my 4 Centurions until then (AH, D, YLW, AL)

As for your Centurion, read this, please:

Piranha Games Inc said:

Intermittent Crashes

• We have identified an issue with Centurion variants that were causing crashes and the fix is confirmed
  
• The Variants causing the issues are CN9-AH(L), CN9-A(NCIX), CN9-D(P)
  
• The Onyx is safe to use

Essentially, it would be better not to use those aforementioned Centurion variants until some possible hotfix or the upcoming December 13 game patch fix the problem.

[AmbidXtrousGNOME]

Matt Newman, on 03 December 2023 - 12:07 AM, said:

We do have database snap shots for sure and source control (perforce) on all our game assets and code.
Rolling back is not impossible but it is not always the best option.

When we first started seeing issues I brought all the lead engineers, Database Admin, QA lead, Web Dev, Infrastructure Network/IT and build engineers together and relayed the situation and the player experience. From there they each set about investigating the behaviors of their respective domains (looking at server logs, checking network routing Etc) so we can find the cause and hopefully the solution as soon as possible. After some time of investigating and cross checking with each other we discuss our findings and our options to fix the problem. In this case the reduction of the overall number of items in the game was what we determined would be the best way forward. It was an easy call for it to be bolt-ons as they made up more than 55% of the item IDs, are cosmetic, and had a clear path to optimize.

Rolling back would have potentially taken longer, carried more risk, and been more work. AND we would still be in the same position where we couldn't add more items to the game without doing the work to remove the bolt-ons.

Hopefully that gives you some insight on that.

All of that makes sense. Thank you for taking the time to answer a nerdy IT question : >

[Mechwarrior2342356]

martian, on 03 December 2023 - 11:02 AM, said:

As for your Centurion, read this, please:

Essentially, it would be better not to use those aforementioned Centurion variants until some possible hotfix or the upcoming December 13 game patch fix the problem.

Is it ONLY the special variants then? I did see an NCIX in one match that wasn't having issues.

Edited by the check engine light, 03 December 2023 - 01:00 PM.

[martian]

the check engine light, on 03 December 2023 - 12:59 PM, said:

Is it ONLY the special variants then? I did see an NCIX in one match that wasn't having issues.

I was quoting what PGI announced.

I can only say that I deployed in CN9-AL about an hour ago in multiple games and I had no problems.

[Mal Bolge]

Wouldnt it be easier to just disable bolt-ons completely while they're trying to fix the problem? It's not like they are essential to the gameplay. I'm pretty sure they loose way more players on not being able to get a match rather than not seeing a cosmetic add-on.

[Horseman]

Besh, on 02 December 2023 - 07:41 AM, said:

So how and why does this erase all my 'Mechs and set my 'Mechbays to 0 for about 15 - 20 mins ?

I suspect your inventory is cached on the backend to avoid too many queries to the underlying database. It's how I would implement it. When it works, it saves some performance, but at the expense of potentially caching faulty responses if the backend is having issues like now.

Besh, on 02 December 2023 - 01:50 PM, said:

Here is a question : How is it anyhow feasable for the "the Server" to need to load "the entire item database" for every match to be able to "crossreference Items" ?

Also known as: one of the associated queries causes or requires a table scan.

Matt Newman, on 02 December 2023 - 03:17 PM, said:

I guess the question then becomes why doesn't the Server have the Data all ready?
Why load all the Item IDS and Data at runtime rather than have that Data present within the server executable?
I don't know the answer to that but I would guess that there is a good reason. Executable size maybe?(I'll check and see if I can find out)

Keeping the data and code separately, so that the data can be updated or adjusted without requiring the code itself to be rebuilt every time. It makes sense. As does not caching it if you want accuracy over performance.

Besh, on 03 December 2023 - 12:18 AM, said:

Preempting "I have no clue how the engine works" ( which is true actually ), I would think it would be more efficient re macthinstance to store Individual player loadouts as a hash under that player's Profile . Would just need to be verified as valid re Item IDs at "save" . Then when [Mech] gets sent into match, the instance just pulls the IDs needed for that specific Match ( instead of loading the entire Item Database into each match instance ) .
But *heh*, what do I know ? I am just trying to understand whats going on .

Sometimes a decision that seemed expedient ten years back during early development isn't very sustainable in the long run. Suffice to say, there's a software product I support that has been running into similar issues when scaling instance size beyond what the dev team originally assumed and tested against.

Viltor, on 03 December 2023 - 06:06 AM, said:

Do I understand this right?
Game is broken, and devs don't know how to fix it?

Very wrong. They have a plan to fix it but it's not something they can just ram through overnight.

the check engine light, on 03 December 2023 - 07:44 AM, said:

While I understand this I also don't really comprehend the idea of "extensive planning and overengineering is bad" or "good enough for now is good enough"

No software company has infinite time or infinite money, they need to bring a working product to the market sooner or later.
Usually companies prioritize things that actively impact the product over fixing potential issues that may occur half a decade down the line... if they even bother testing against such highly improbable scenarios in the first place.

Thunderbuddy, on 03 December 2023 - 10:05 AM, said:

As a brand new player with 1 mech, it is very offputting to have my mech locked and not be able to locate matches. This game is way too old to be having open beta level issues.

It's not an open beta kind of issue, it's a ten year old piece of tech debt come to roost.

Edited by Horseman, 03 December 2023 - 01:52 PM.

[Mechwarrior2342356]

CN9-AH(L) on team in Bearclaw, crashed out then crashed me out at end of match.

At this point dropping in that mech is borderline griefing. Please don't do it until this is fixed.

[Matt Newman]

Mal Bolge, on 03 December 2023 - 01:39 PM, said:

Wouldnt it be easier to just disable bolt-ons completely while they're trying to fix the problem? It's not like they are essential to the gameplay. I'm pretty sure they loose way more players on not being able to get a match rather than not seeing a cosmetic add-on.

If there was a disable bolt ons button I would have pressed that button so fast XD

Horseman, on 03 December 2023 - 01:40 PM, said:

I suspect your inventory is cached on the backend to avoid too many queries to the underlying database. It's how I would implement it. When it works, it saves some performance, but at the expense of potentially caching faulty responses if the backend is having issues like now.
Also known as: one of the associated queries causes or requires a table scan.
Keeping the data and code separately, so that the data can be updated or adjusted without requiring the code itself to be rebuilt every time. It makes sense. As does not caching it if you want accuracy over performance.

Sometimes a decision that seemed expedient ten years back during early development isn't very sustainable in the long run. Suffice to say, there's a software product I support that has been running into similar issues when scaling instance size beyond what the dev team originally assumed and tested against.

Very wrong. They have a plan to fix it but it's not something they can just ram through overnight.


No software company has infinite time or infinite money, they need to bring a working product to the market sooner or later.
Usually companies prioritize things that actively impact the product over fixing potential issues that may occur half a decade down the line... if they even bother testing against such highly improbable scenarios in the first place.


It's not an open beta kind of issue, it's a ten year old piece of tech debt come to roost.

Tell me you work in tech without telling me you work in tech

the check engine light, on 03 December 2023 - 01:56 PM, said:

CN9-AH(L) on team in Bearclaw, crashed out then crashed me out at end of match.

At this point dropping in that mech is borderline griefing. Please don't do it until this is fixed.

Let’s hope it’s not intentional >

[Split9]

I know it's an known issue, so just another voice adding to the noise here but 2 games in an hour. Unplayable right now

Edited by Split9, 03 December 2023 - 03:38 PM.

[Rebel Ace Fryslan]

Don't use bolt-ons, yet keep getting disconnected.

And please remove the tier system, it'just isn't representative of skill and teamwork.

[RogueGarou]

It has probably been reported, I did not read all of the comments in the thread, but two matches in a row last night when I had the new Onyx Mech selected. One match with the Onyx worked properly. Probably coincidental. I was in a group with one other player. We had several other matches with no problem using other Mechs. I think we had 6 matches with a Flea and a Black Lanner with no errors. I changed to use the Onyx and it worked for the first match but the next two attempts failed. No idea what the other 2 Mechs in those matches were since the crash happened before the screen showing the teams loaded. Third attempt and MWO crashed for both of us so we called it a night.

It is just anecdotal that these all happened while trying to use the Onyx. It was the only common factor in the three matches where we had problems.

One thing that was odd after the patch was that the Gorewing was then listed as Invalid. I tested what the problem was by removing components one by one, weapons first, ammo, equipment, etc. It turned out to be the armor that made it invalid. Removing all of the armor and then adding back the same amount of armor corrected the issue. I also had something similar that had been in the Mech inventory for a few months that I corrected in the same way. The Nova Prime Invasion Mech had been showing the image of a Timber Wolf for a long time but I had not played that Mech so it was just an annoyance. Removing all armor and adding it back corrected that and the Mech shows the correct image in the inventory now. I am just putting this out there in case anyone else has had the same issues and is looking for a fix.

Good luck with finding the root cause of the issues with the Centurion Mechs and the game hiccups. From my personal observation, selecting a different Mech allows a match to be started so it isn't apocalyptic.. It is annoying and weird but it has a quick work around.

EDIT:
Well, it seems to be much worse tonight than on previous nights. So far, 10 for 10 drops without a single successful match. 9 different Mechs in case there was an issue with the Mech. Different classes, different types and different chassis. Light, Medium, Heavy, Clan, IS, Platinum, standard, Hero, etc.

At this point, I think the game is actually unplayable because, well, it cannot be played at all. I don't want to be a complainer without cause but tonight, it seems to be justifiable.

Edited by RogueGarou, 03 December 2023 - 06:35 PM.

[Voice of Kerensky]

МВО, не болей.
MWO, don't be sick.

Edited by Voice of Kerensky, 03 December 2023 - 04:11 PM.

[Tywren]

With this being a known, widespread issue, PGI really needs to lock the ELO until the game is once again stable. Loosing rank due to the server timing out is beyond player control and shouldn't be something they are penalized for.

[Wooky1]

been playing since 2012 been a couple of rough spots over the years, but this is stupid tried for a 1/2 hr to get a game mech locked every time. i will not be back until it fixed and I'm sure its been fixed completely i mean you just couldn't roll it back