Jump to content

Mwo Status Update And Known Issues - Update 2 - Dec 1St


305 replies to this topic

#221 martian

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 8,574 posts

Posted 03 December 2023 - 10:55 AM

View PostMaxx31, on 03 December 2023 - 10:39 AM, said:

What about any Premium Time that was burned due to the issues? I would say I agree there is not a 100% allocation (games that do run, sleep, eating, etc).


Read this, please:

Piranha Games Inc. said:

Once the fix is deployed, we plan to host an event with compensation as a gesture of appreciation.

The compensation event and all the Holiday events will not be posted or started until the patch is live, and we can confirm on the live environment that everything is back to normal.


#222 martian

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 8,574 posts

Posted 03 December 2023 - 11:02 AM

View Postthe check engine light, on 03 December 2023 - 10:29 AM, said:

Ten days with the ability to occasionally get matches until then isn't terrible though I'm kind of cheesed off that I can't play any of my 4 Centurions until then (AH, D, YLW, AL)


As for your Centurion, read this, please:

Piranha Games Inc said:

Intermittent Crashes
  • We have identified an issue with Centurion variants that were causing crashes and the fix is confirmed
  • The Variants causing the issues are CN9-AH(L), CN9-A(NCIX), CN9-D(P)
  • The Onyx is safe to use Posted Image

Essentially, it would be better not to use those aforementioned Centurion variants until some possible hotfix or the upcoming December 13 game patch fix the problem.

#223 AmbidXtrousGNOME

    Member

  • PipPipPipPipPip
  • The Cyber Warrior
  • The Cyber Warrior
  • 107 posts

Posted 03 December 2023 - 11:57 AM

View PostMatt Newman, on 03 December 2023 - 12:07 AM, said:


We do have database snap shots for sure and source control (perforce) on all our game assets and code.
Rolling back is not impossible but it is not always the best option.

When we first started seeing issues I brought all the lead engineers, Database Admin, QA lead, Web Dev, Infrastructure Network/IT and build engineers together and relayed the situation and the player experience. From there they each set about investigating the behaviors of their respective domains (looking at server logs, checking network routing Etc) so we can find the cause and hopefully the solution as soon as possible. After some time of investigating and cross checking with each other we discuss our findings and our options to fix the problem. In this case the reduction of the overall number of items in the game was what we determined would be the best way forward. It was an easy call for it to be bolt-ons as they made up more than 55% of the item IDs, are cosmetic, and had a clear path to optimize.

Rolling back would have potentially taken longer, carried more risk, and been more work. AND we would still be in the same position where we couldn't add more items to the game without doing the work to remove the bolt-ons.

Hopefully that gives you some insight on that.


All of that makes sense. Thank you for taking the time to answer a nerdy IT question : >

#224 martian

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 8,574 posts

Posted 03 December 2023 - 01:06 PM

View Postthe check engine light, on 03 December 2023 - 12:59 PM, said:

Is it ONLY the special variants then? I did see an NCIX in one match that wasn't having issues.

I was quoting what PGI announced.

I can only say that I deployed in CN9-AL about an hour ago in multiple games and I had no problems.

#225 Mal Bolge

    Member

  • PipPipPipPipPip
  • 104 posts

Posted 03 December 2023 - 01:39 PM

Wouldnt it be easier to just disable bolt-ons completely while they're trying to fix the problem? It's not like they are essential to the gameplay. I'm pretty sure they loose way more players on not being able to get a match rather than not seeing a cosmetic add-on.

#226 Horseman

    Member

  • PipPipPipPipPipPipPipPipPip
  • The Mercenary
  • The Mercenary
  • 4,738 posts
  • LocationPoland

Posted 03 December 2023 - 01:40 PM

View PostBesh, on 02 December 2023 - 07:41 AM, said:

So how and why does this erase all my 'Mechs and set my 'Mechbays to 0 for about 15 - 20 mins ?
I suspect your inventory is cached on the backend to avoid too many queries to the underlying database. It's how I would implement it. When it works, it saves some performance, but at the expense of potentially caching faulty responses if the backend is having issues like now.

View PostBesh, on 02 December 2023 - 01:50 PM, said:

Here is a question : How is it anyhow feasable for the "the Server" to need to load "the entire item database" for every match to be able to "crossreference Items" ?
Also known as: one of the associated queries causes or requires a table scan.

View PostMatt Newman, on 02 December 2023 - 03:17 PM, said:

I guess the question then becomes why doesn't the Server have the Data all ready?
Why load all the Item IDS and Data at runtime rather than have that Data present within the server executable?
I don't know the answer to that but I would guess that there is a good reason. Executable size maybe?(I'll check and see if I can find out)
Keeping the data and code separately, so that the data can be updated or adjusted without requiring the code itself to be rebuilt every time. It makes sense. As does not caching it if you want accuracy over performance.

View PostBesh, on 03 December 2023 - 12:18 AM, said:

Preempting "I have no clue how the engine works" ( which is true actually ), I would think it would be more efficient re macthinstance to store Individual player loadouts as a hash under that player's Profile . Would just need to be verified as valid re Item IDs at "save" . Then when [Mech] gets sent into match, the instance just pulls the IDs needed for that specific Match ( instead of loading the entire Item Database into each match instance ) .
But *heh*, what do I know ? I am just trying to understand whats going on .

Sometimes a decision that seemed expedient ten years back during early development isn't very sustainable in the long run. Suffice to say, there's a software product I support that has been running into similar issues when scaling instance size beyond what the dev team originally assumed and tested against.

View PostViltor, on 03 December 2023 - 06:06 AM, said:

Do I understand this right?
Game is broken, and devs don't know how to fix it?

Very wrong. They have a plan to fix it but it's not something they can just ram through overnight.

View Postthe check engine light, on 03 December 2023 - 07:44 AM, said:

While I understand this I also don't really comprehend the idea of "extensive planning and overengineering is bad" or "good enough for now is good enough"

No software company has infinite time or infinite money, they need to bring a working product to the market sooner or later.
Usually companies prioritize things that actively impact the product over fixing potential issues that may occur half a decade down the line... if they even bother testing against such highly improbable scenarios in the first place.

View PostThunderbuddy, on 03 December 2023 - 10:05 AM, said:

As a brand new player with 1 mech, it is very offputting to have my mech locked and not be able to locate matches. This game is way too old to be having open beta level issues.

It's not an open beta kind of issue, it's a ten year old piece of tech debt come to roost.

Edited by Horseman, 03 December 2023 - 01:52 PM.


#227 Matt Newman

    Live Ops Manager

  • Developer
  • Developer
  • 1,055 posts

Posted 03 December 2023 - 02:34 PM

View PostMal Bolge, on 03 December 2023 - 01:39 PM, said:

Wouldnt it be easier to just disable bolt-ons completely while they're trying to fix the problem? It's not like they are essential to the gameplay. I'm pretty sure they loose way more players on not being able to get a match rather than not seeing a cosmetic add-on.


If there was a disable bolt ons button I would have pressed that button so fast XD

View PostHorseman, on 03 December 2023 - 01:40 PM, said:

I suspect your inventory is cached on the backend to avoid too many queries to the underlying database. It's how I would implement it. When it works, it saves some performance, but at the expense of potentially caching faulty responses if the backend is having issues like now.
Also known as: one of the associated queries causes or requires a table scan.
Keeping the data and code separately, so that the data can be updated or adjusted without requiring the code itself to be rebuilt every time. It makes sense. As does not caching it if you want accuracy over performance.

Sometimes a decision that seemed expedient ten years back during early development isn't very sustainable in the long run. Suffice to say, there's a software product I support that has been running into similar issues when scaling instance size beyond what the dev team originally assumed and tested against.

Very wrong. They have a plan to fix it but it's not something they can just ram through overnight.


No software company has infinite time or infinite money, they need to bring a working product to the market sooner or later.
Usually companies prioritize things that actively impact the product over fixing potential issues that may occur half a decade down the line... if they even bother testing against such highly improbable scenarios in the first place.


It's not an open beta kind of issue, it's a ten year old piece of tech debt come to roost.



Tell me you work in tech without telling me you work in tech ;)

View Postthe check engine light, on 03 December 2023 - 01:56 PM, said:

CN9-AH(L) on team in Bearclaw, crashed out then crashed me out at end of match.

At this point dropping in that mech is borderline griefing. Please don't do it until this is fixed.


Let’s hope it’s not intentional >:(

#228 Split9

    Rookie

  • Bridesmaid
  • Bridesmaid
  • 3 posts
  • LocationWA

Posted 03 December 2023 - 03:37 PM

I know it's an known issue, so just another voice adding to the noise here but 2 games in an hour. Unplayable right now

Edited by Split9, 03 December 2023 - 03:38 PM.


#229 Rebel Ace Fryslan

    Member

  • PipPipPipPipPipPip
  • FP Veteran - Beta 1
  • 445 posts
  • LocationAd Astra

Posted 03 December 2023 - 03:40 PM

Don't use bolt-ons, yet keep getting disconnected.

And please remove the tier system, it'just isn't representative of skill and teamwork.

#230 RogueGarou

    Member

  • Pip
  • Wrath
  • Wrath
  • 16 posts

Posted 03 December 2023 - 03:45 PM

It has probably been reported, I did not read all of the comments in the thread, but two matches in a row last night when I had the new Onyx Mech selected. One match with the Onyx worked properly. Probably coincidental. I was in a group with one other player. We had several other matches with no problem using other Mechs. I think we had 6 matches with a Flea and a Black Lanner with no errors. I changed to use the Onyx and it worked for the first match but the next two attempts failed. No idea what the other 2 Mechs in those matches were since the crash happened before the screen showing the teams loaded. Third attempt and MWO crashed for both of us so we called it a night.

It is just anecdotal that these all happened while trying to use the Onyx. It was the only common factor in the three matches where we had problems.

One thing that was odd after the patch was that the Gorewing was then listed as Invalid. I tested what the problem was by removing components one by one, weapons first, ammo, equipment, etc. It turned out to be the armor that made it invalid. Removing all of the armor and then adding back the same amount of armor corrected the issue. I also had something similar that had been in the Mech inventory for a few months that I corrected in the same way. The Nova Prime Invasion Mech had been showing the image of a Timber Wolf for a long time but I had not played that Mech so it was just an annoyance. Removing all armor and adding it back corrected that and the Mech shows the correct image in the inventory now. I am just putting this out there in case anyone else has had the same issues and is looking for a fix.

Good luck with finding the root cause of the issues with the Centurion Mechs and the game hiccups. From my personal observation, selecting a different Mech allows a match to be started so it isn't apocalyptic.. It is annoying and weird but it has a quick work around.

EDIT:
Well, it seems to be much worse tonight than on previous nights. So far, 10 for 10 drops without a single successful match. 9 different Mechs in case there was an issue with the Mech. Different classes, different types and different chassis. Light, Medium, Heavy, Clan, IS, Platinum, standard, Hero, etc.

At this point, I think the game is actually unplayable because, well, it cannot be played at all. :) I don't want to be a complainer without cause but tonight, it seems to be justifiable.

Edited by RogueGarou, 03 December 2023 - 06:35 PM.


#231 Voice of Kerensky

    Member

  • PipPipPipPipPipPipPip
  • Ace Of Spades
  • Ace Of Spades
  • 512 posts

Posted 03 December 2023 - 04:09 PM

МВО, не болей.
MWO, don't be sick.
Posted Image

Edited by Voice of Kerensky, 03 December 2023 - 04:11 PM.


#232 Tywren

    Member

  • PipPipPipPipPipPip
  • Ace Of Spades
  • 277 posts

Posted 03 December 2023 - 04:27 PM

With this being a known, widespread issue, PGI really needs to lock the ELO until the game is once again stable. Loosing rank due to the server timing out is beyond player control and shouldn't be something they are penalized for.

#233 Wooky1

    Member

  • Pip
  • Ace Of Spades
  • Ace Of Spades
  • 10 posts

Posted 03 December 2023 - 05:10 PM

been playing since 2012 been a couple of rough spots over the years, but this is stupid tried for a 1/2 hr to get a game mech locked every time. i will not be back until it fixed and I'm sure its been fixed completely i mean you just couldn't roll it back

#234 Tarl Cabot

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • Tai-sho
  • Tai-sho
  • 7,793 posts
  • LocationImperial City, Luthien - Draconis Combine

Posted 03 December 2023 - 05:20 PM

View PostTywren, on 03 December 2023 - 04:27 PM, said:

With this being a known, widespread issue, PGI really needs to lock the ELO until the game is once again stable. Loosing rank due to the server timing out is beyond player control and shouldn't be something they are penalized for.


With the no server found, players PSR are not being impacted. Notice that your Multiplier doesnt change :) And we left Elo behind many years ago when PGI went to PSR/Tier settings.

#235 Remover of Obstacles

    Member

  • PipPipPipPipPipPipPip
  • Fire
  • Fire
  • 568 posts

Posted 03 December 2023 - 05:51 PM

100% failure rate today.

When will this be fixed?

Edited by Remover of Obstacles, 03 December 2023 - 05:51 PM.


#236 avenger cannon

    Member

  • PipPipPip
  • The Clamps
  • The Clamps
  • 75 posts

Posted 03 December 2023 - 06:12 PM

Please fix this, remove Bearclaw 2 until it can be fixed ?

#237 King Yellowman

    Member

  • Pip
  • The Tyrant
  • The Tyrant
  • 12 posts

Posted 03 December 2023 - 08:11 PM

this is pure madness

#238 IndgoMan

    Member

  • PipPipPip
  • CS 2024 Top 25
  • CS 2024 Top 25
  • 76 posts
  • Locationohio

Posted 03 December 2023 - 08:29 PM

hope this get fixed, love the game i want to get back to enjoying it.

#239 Void Angel

    Member

  • PipPipPipPipPipPipPipPipPipPip
  • The Marauder
  • The Marauder
  • 7,185 posts
  • LocationParanoiaville

Posted 03 December 2023 - 08:42 PM

View PostTywren, on 03 December 2023 - 04:27 PM, said:

With this being a known, widespread issue, PGI really needs to lock the ELO until the game is once again stable. Loosing rank due to the server timing out is beyond player control and shouldn't be something they are penalized for.


"Elo" is not an acronym - also PSR isn't quite Elo, but we know what you mean.

View PostMatt Newman, on 03 December 2023 - 02:34 PM, said:

Tell me you work in tech without telling me you work in tech Posted Image


What fascinates me - who does not work in tech - is how the problem started up gradually, then moved. It's like you have a memory leak as a kind of athletic infection...

How does that work?

#240 Ghost of Ligeia Mare

    Rookie

  • Ironclad
  • Ironclad
  • 8 posts

Posted 03 December 2023 - 09:08 PM

Why aren't you just using DynamoDB with a cloned database for public testing? It's cheap, man.





2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users