326 replies to this topic

[Besh]

Tarl Cabot, on 02 December 2023 - 11:43 PM, said:

It has to pull/load that info for each instance, else the main servers would choke. And based on the information provided, the primary issue is the a combination of the database size + unpacking that data when called, the resources of that instance are exceeded. Else all drops would be kicking everyone back to the mechbay.

Preempting "I have no clue how the engine works" ( which is true actually ), I would think it would be more efficient re macthinstance to store Individual player loadouts as a hash under that player's Profile . Would just need to be verified as valid re Item IDs at "save" . Then when [Mech] gets sent into match, the instance just pulls the IDs needed for that specific Match ( instead of loading the entire Item Database into each match instance ) .

But *heh*, what do I know ? I am just trying to understand whats going on .

Edited by Besh, 03 December 2023 - 01:26 AM.

[Matt Newman]

Besh, on 02 December 2023 - 10:13 PM, said:

I was/am just not understanding why Server needs to load entire Database at Lobby/Matchcreation . I would have thought every 'Mech/Loadout, including boltons etc. is stored as is, and when used ( sent into Match ), Server just needs to pull those IDs that are being used by the 24 'Mechs in that particular Match from the Database .

To stay in metaphor ( I was working in kitchens ) : When an order for somethng comes in, I don't ned to recall ALL the ingredients we have stored . All I need to know to prepare that meal/dish are the specific ingredients for whats has been ordered ( bun, patty, sauces/drsseing, garnish etc .) . When a veggíe burger is ordered, I dont need to know I could make it with either a ToFu/Legumes/Mushroom Patty . I read its ordered with ToFu patty, and I grab exactly that from storage and use it.

Additionally, having exceeded the max# of Items in context to me would mean either
buffer overflow
or
database overflow .

And this lead me to thinking what Eiji mentioned :



The intermittence of the Issue to me just seemed to point to a Memory allocation Issue rather than a Database Issue . Or maybe even a HW problem XD .

Anyhow . Thanks . Transparency, time spent to check this Thread and answer individual questions - esp. at weekend! - are much appreciated .

p.s.: Some humblebrag to explain why I am interested in understanding the Problem . I was employed at H&P as Support Tech Networking/Servers & Storage in one of their international Centers a few decades ago . While I haven't worked in the field since, and ofc my "knowledge" ( if it can even be called that ) is more than rusty, asking myself "Whats happening here ?" when looking at a Tech problem still is some kind of autoroutine .

Those are fair questions and I have to admit I am not an expert! The presenting behaviors in the software do feel like memory overruns but I am assured by people way smarter than I am that it is the overall number of items in the database that is causing these issues. The Intermittence AFAIK can be chalked up to large data transmission?... lets pretend our database is a Tweet on twitter and we have been keeping every tweet under the Character limit of 300 but somehow we just start tweeting at 310 characters and Elon Musk is mad at us!? (I am getting further out of my knowledge comfort zone must switch back to food based metaphor!!)

Also I am glad you like my Kitchen Metaphor I will say that yes as humans we don't have to load every ingredient when we are making a specific dish! BUT think about how many dishes you know how to make! Or how you know what spices and flavors to add to get a specific flavor its almost like you DO have a Database in there right?

[w0qj]

Guys, let's not be so hard on MWO...

1. IF MWO programmers had the foresight and resources to program for contingencies 10+ years into the future, it would be called

Over-engineered

ie: cost of this would be prohibitive, cost=high means less F2P players, causing cost to go even higher, to the point that MWO does not make any money, and servers are shutdown as a business decision, and well before the 10+ year mark.

Many online games don't last over 3 years, so why should MWO plan for 10+ years into the future?

2. What we are seeing is there's been a dramatic rise in number of mechs in the last 2-3 years, and the increasing number of bolt-ons were compounded by this rise in number of mechs as well.

(a) What is the answer for 1x2x3x4x5x6x7x8x9x10 = ?

(b ) What is the answer for 1x2x3x4x5x6x7x8x9x10x11x12x13x14x15 = ?

The difference in magnitude between answers to (a) vs (b ) may surprise you.

I believe this is one of the factor leading up to this approaching/exceeding maximum database count issue, thus causing this great server instability.

(That said, I've succeeded in leveling up a number of my mechs, got my WHK free mech for Nov, etc.)

Edited by w0qj, 03 December 2023 - 12:39 AM.

[Matt Newman]

Tarl Cabot, on 03 December 2023 - 12:12 AM, said:

Matt,
Back tracking on my previous post, thus asking for clarification. Thus a question on the no server found. Is just that player affected or all 24 players for that actual server instance? Just from the lack of PSR movement, it would appear to be that server instance.

The entire match fails. No one is in the match. No PSR changes as the match does not officially spin up.
Unfortunately the Mechs still get locked as that happens when the match group is created, the map and mode are chosen, and all the information is sent to the server.

PS going to bed now

martian, on 03 December 2023 - 12:08 AM, said:

Maybe with the company, but not working on MWO - at least that is how I understood the Russ Bullock's words when he described the current MWO status. Do I get it right?

Ah I see what you mean. Yes on other projects but I can always call on them in times such as these!

[martian]

Matt Newman, on 03 December 2023 - 12:42 AM, said:

Ah I see what you mean. Yes on other projects but I can always call on them in times such as these!

That is good to know.

[Besh]

Matt Newman, on 03 December 2023 - 12:33 AM, said:

Those are fair questions and I have to admit I am not an expert! The presenting behaviors in the software do feel like memory overruns but I am assured by people way smarter than I am that it is the overall number of items in the database that is causing these issues. The Intermittence AFAIK can be chalked up to large data transmission?... lets pretend our database is a Tweet on twitter and we have been keeping every tweet under the Character limit of 300 but somehow we just start tweeting at 310 characters and Elon Musk is mad at us!? (I am getting further out of my knowledge comfort zone must switch back to food based metaphor!!)

Also I am glad you like my Kitchen Metaphor I will say that yes as humans we don't have to load every ingredient when we are making a specific dish! BUT think about how many dishes you know how to make! Or how you know what spices and flavors to add to get a specific flavor its almost like you DO have a Database in there right?

Well, IT in principle is/was modelled after how it is/was understood at times how brain/mind works .

Looking at some items involved in creating the Issue...

ItemDatabase : Assumed it has an index limit, this limit is ( should have been ) known and observed at all times for obvious reasons;
Data transfer : it seems transferring the entire ItemDB for each Match isn't relly efficient ;
Load: for both, Database and MatchInstances, load is (probably unnecessarily) high when having to transfer/load the entire ItemDB for each Match .
HW considerations : each instance having to load the entire ItemDB does not lead to unnecessary bloat in HW requirements, making the System more vulnerable to simple HW failures ?

In short : Interesting ! Designing this Game 12 years ago, one would do it with the prospect of growth in mind . Growth for MW:O inevitably means "More Matches" ( amongst other things ) . So one aspect I would go ahead and make as resource efficient as possible is how the System handles increasing number of Matches . But (honestly) what do I know ? Ofc hindsight is 20/20, and I am basically somewhat trying to reverse engineer the thought processes from 12 years ago without knowing anything about the constraints an challenges that had to be dealt with . Which obviously is an endeavour stretching the scope of this Thread a bit .


Anyhow . Thanks again for spending time with us at the weekend .

Sleep well .

Edited by Besh, 03 December 2023 - 01:19 AM.

[Besh]

w0qj, on 03 December 2023 - 12:35 AM, said:

Guys, let's not be so hard on MWO...

1. IF MWO programmers had the foresight and resources to program for contingencies 10+ years into the future, it would be called

Over-engineered

ie: cost of this would be prohibitive, cost=high means less F2P players, causing cost to go even higher, to the point that MWO does not make any money, and servers are shutdown as a business decision, and well before the 10+ year mark.

Many online games don't last over 3 years, so why should MWO plan for 10+ years into the future?

2. What we are seeing is there's been a dramatic rise in number of mechs in the last 2-3 years, and the increasing number of bolt-ons were compounded by this rise in number of mechs as well.

(a) What is the answer for 1x2x3x4x5x6x7x8x9x10 = ?

(b ) What is the answer for 1x2x3x4x5x6x7x8x9x10x11x12x13x14x15 = ?

The difference in magnitude between answers to (a) vs (b ) may surprise you.

I believe this is one of the factor leading up to this approaching/exceeding maximum database count issue, thus causing this great server instability.

(That said, I've succeeded in leveling up a number of my mechs, got my WHK free mech for Nov, etc.)

Exploding Database due to constantly adding Items at sometimes rapid pace can be a challenge in itself I guess .
Having to load that entire DataBase for each Match individiually in context of a growing online teambased PvP shooter sounds like the kind of nightmare I would not wish onto enemies .

p.s.: I am not trying to be hard. I am trying to understand. I realize language/culture differences may lead to misunderstandings . Like, me being german, I am very much used to putting stuff in " " when quoting, and I find myself doing this a lot on forums/boards instead of hitting the quote button then editing, since its just faster - esp. when quoting phrases instead of an entire paragraph . But me quoting what someone wrote in " " may make some think I do it because I doubt what was written .

Edited by Besh, 03 December 2023 - 01:18 AM.

[Hagen Zwosta]

Are the servers completely gone now? 4 locked mechs in a row and counting.
edit: suddenly its fine.

Edited by Hagen Zwosta, 03 December 2023 - 02:11 AM.

[Steel Mike Eels]

So I have a question:

Is this an issue with the actual database software in use (ie whatever is being used cannot handle that many records?) or is this an issue with the throughput of raw data when matches are queued up?

[tenchugecko]

Dear Guys

You should be very hard to "MWO" about this fuckup.

I any IT Project, you will calculate the sizes and frequencies of data, and find out maximum Values Systems/Servers/Clients can handle

Instead they just misused the item system in a hack for an insignificant feature like boltons.
It's not OverEngineering, it's just plain 1st semester informatics to know not to do that.
And just like the GTA Item Cross Reference bug, this **** was smelling from afar -
with MWO Startup Times scaling negatively with avaiable computing power through the years.

All of this, while having no option to disable bolt-ons temporarily.

But what do i know as a Software Project Manager.

(PS. 3 times disco today.)

[Viltor]

Do I understand this right?
Game is broken, and devs don't know how to fix it?

[Besh]

Steel Mike Eels, on 03 December 2023 - 03:22 AM, said:

So I have a question:

Is this an issue with the actual database software in use (ie whatever is being used cannot handle that many records?) or is this an issue with the throughput of raw data when matches are queued up?

...or maybe a memory allocation issue, i.e. db *suddenly* having become too big for individual ones OR the total # of MatchInstances simultaneously being run on the available resources, as it seems like every instance has to load the entire db . If that, could be HW/SW/combination .

A "simply indexing" ( not sure I understand what ItemDB for MW:O does, and how, hence " " ) DB *should* be expandable .

Edited by Besh, 03 December 2023 - 06:20 AM.

[ExoForce]

Long time ago somebody posted here somewhere the detailed comparison between cryengine and other game engines in terms of handling in-game objects. IIRC in simple words it was cryengine (as implemented in this game) is not good for the hit registration of multiple sub-objects on main item.

Elders are remembering what clan Lrms and MGs did to servers.
IIRC the internal hot-fix was that servers where dropping half of the damage (or not registering half of the shoots).
Official explanation (after quite some time) was like *server hitreg timer was wrongly configured*...

Praise the cryengine (as implemented in this game).

Edited by ExoForce, 03 December 2023 - 07:11 AM.

[martian]

Viltor, on 03 December 2023 - 06:06 AM, said:

Do I understand this right?
Game is broken, and devs don't know how to fix it?

No, you do not understand it right.

Read this update, please: Mwo Status Update And Known Issues - Update 2 - Dec 1St

[Mechwarrior2342356]

w0qj, on 03 December 2023 - 12:35 AM, said:

Guys, let's not be so hard on MWO...

1. IF MWO programmers had the foresight and resources to program for contingencies 10+ years into the future, it would be called

Over-engineered

ie: cost of this would be prohibitive, cost=high means less F2P players, causing cost to go even higher, to the point that MWO does not make any money, and servers are shutdown as a business decision, and well before the 10+ year mark.

Many online games don't last over 3 years, so why should MWO plan for 10+ years into the future?

2. What we are seeing is there's been a dramatic rise in number of mechs in the last 2-3 years, and the increasing number of bolt-ons were compounded by this rise in number of mechs as well.

(a) What is the answer for 1x2x3x4x5x6x7x8x9x10 = ?

(b ) What is the answer for 1x2x3x4x5x6x7x8x9x10x11x12x13x14x15 = ?

The difference in magnitude between answers to (a) vs (b ) may surprise you.

I believe this is one of the factor leading up to this approaching/exceeding maximum database count issue, thus causing this great server instability.

(That said, I've succeeded in leveling up a number of my mechs, got my WHK free mech for Nov, etc.)

While I understand this I also don't really comprehend the idea of "extensive planning and overengineering is bad" or "good enough for now is good enough"

[Saved By The Bell]

Use oceanic server, while its still somehow alive.

Dont blame the PGI team, because its not help anyone.

[Mechwarrior2342356]

Saved By The Bell, on 03 December 2023 - 08:21 AM, said:

Use oceanic server, while its still somehow alive.

Dont blame the PGI team, because its not help anyone.

yeah this is pretty much on past members leaving a mess for later members to clean up after due to neglect and miserable choices regarding corners cut and time spent (and poor hand off with known information about the software's intricacies)

Can relate given current job, pretty strongly actually

Edited by the check engine light, 03 December 2023 - 09:09 AM.

[ScrapIron Prime]

Viltor, on 03 December 2023 - 06:06 AM, said:

Do I understand this right?
Game is broken, and devs don't know how to fix it?

Incorrect. They know how to fix it, the issue is the hard choices required to do so.

The core engine uses a database circa 2003. Older databases had a much lower maximum record count than databases today, so the task before them is to reduce the record count in the database by enough to not just support everything in the game now, but leave extra space for everything they might add to the game in a couple years.

they’ve chosen to reclassify and combine bolt ons so that multiple kinds can use a single database entry, rather than simply deleting content. This is not simple, but they know what has to be done. Expect certain bolt ins to be goofed up for a while as they work out the fine details.

the other option would be to upgrade the database engine of the game… but that’s making a whole new game engine…and that’s totally out of scope.

Devs… as a guy who’s been a computer engineer for 32 years, I’ve been where you are now. You’ve got this.

[Thunderbuddy]

As a brand new player with 1 mech, it is very offputting to have my mech locked and not be able to locate matches. This game is way too old to be having open beta level issues.

[w0qj]

A warm welcome to MWO!

Here's how to get your free 2nd mech, if you input this redeem code immediately (time-limited offer!):

Redeem Code:
CS23-EQV5JDX

Use above code after logging into your account: https://mwomercs.com/profile/redeem

- - - - - - - - - - -
Source:
https://mwomercs.com...-mech-from-cs23

Thunderbuddy, on 03 December 2023 - 10:05 AM, said:

As a brand new player with 1 mech, it is very offputting to have my mech locked and not be able to locate matches. This game is way too old to be having open beta level issues.