305 replies to this topic

[w0qj]

I feel that perhaps MWO should consider to give everyone "one(1)" email coupon** for a 50% Off discount on any one(1) single purchase. Keeps players looking at all the packages that they want, have to make a hard choice on which one to buy at 50% Off! Then all is forgiven

Note: '**' This email coupon would expire 31-Jan-2024 ?

Matt Newman, on 02 December 2023 - 03:17 PM, said:

It does make sense when you consider everything you bring into the match is Data. and the Server needs to know what that Data represents.

...Right now we seem to have exceeded the total number of possible ingredients in our kitchen, the Cooks can't prepare the meal in time due to the overhead of loading the ingredient list to build the meal correctly. Even if you come in to order a burger its still loading the full list of possible ingredients to assemble the burger correctly.

I guess the question then becomes why doesn't the Server have the Data all ready?
Why load all the Item IDS and Data at runtime rather than have that Data present within the server executable?
I don't know the answer to that but I would guess that there is a good reason. Executable size maybe?(I'll check and see if I can find out)

Edited by w0qj, 02 December 2023 - 06:47 PM.

[Matt Newman]

At this point "rolling back" the Nov patch would be more work than staying the course with the fix that is in the pipeline for next week. ( I hope to be posting Monday to let you know when the patch will deploy)

I totally understand that this is a frustrating experience. Believe me when I say if there was a better or faster option we would be doing it.

And just so folks know, the bolt-ons are not the issue per se, the total number of items in the database is. Bolt-ons just happen to have the most Item IDs and present the best option for reducing the over all item count in the Database.

[Eiji Sixfinger]

I'm confused as to if it's the total number of items in the database, not what we actual are using/own, why is it so intermittent?
And if it's a timeout, why wasn't increasing the timeout wait time a temporary option?

[AmbidXtrousGNOME]

Matt Newman, on 02 December 2023 - 06:54 PM, said:

At this point "rolling back" the Nov patch would be more work than staying the course with the fix that is in the pipeline for next week. ( I hope to be posting Monday to let you know when the patch will deploy)

I totally understand that this is a frustrating experience. Believe me when I say if there was a better or faster option we would be doing it.

And just so folks know, the bolt-ons are not the issue per se, the total number of items in the database is. Bolt-ons just happen to have the most Item IDs and present the best option for reducing the over all item count in the Database.

Hey Matt,

does PGI have the means to snap-shot servers and restore a server to a date-in-time via image? I'm speaking from a networking understanding of how I do server backup/restores at my job so I'm going out on a limb and assuming the same could be done for gaming servers since (assuming again) there really there isn't much of a difference from a game server and a high traffic webserver.

Maybe that's what you mean by "rolling back" the November patch or maybe it's an actual uninstall process instead of a restore-state kind of thing. Just and idea.

Thanks for the openness about the issues instead of just giving the classic "It's being worked on" kind of BS.

[CrustyMech]

Matt Newman, on 02 December 2023 - 06:54 PM, said:

At this point "rolling back" the Nov patch would be more work than staying the course with the fix that is in the pipeline for next week. ( I hope to be posting Monday to let you know when the patch will deploy)

I totally understand that this is a frustrating experience. Believe me when I say if there was a better or faster option we would be doing it.

And just so folks know, the bolt-ons are not the issue per se, the total number of items in the database is. Bolt-ons just happen to have the most Item IDs and present the best option for reducing the over all item count in the Database.

Thanks for the update. All of the old tech geeks understand the issue and how hard it is to fix in a timely fashion. It's a good thing this game has staying power... must be a Canuck thing

[Strelok7]

Are they deleting some bolt-ons, did I understand correctly?

Edited by Strelok7, 02 December 2023 - 08:54 PM.

[Strelok7]

Matt Newman, on 02 December 2023 - 03:17 PM, said:

It does make sense when you consider everything you bring into the match is Data. and the Server needs to know what that Data represents.

Mechs have Item IDs
Armor Has an Item IDs
Infrastructure has an Item IDs
Engine have Item IDs
Weapons have Item IDs
Ammo Has Item IDs
Decals Have Item IDs
Cockpit Items Have Item IDs


So we send the Player ID and Mech ID and all the loadout data which includes everything that is attached to that mech which are all Item IDs. Send that all to a Server which has to understand that Data Structure and properties and behaviors associated with those Item IDs.

The server needs to know what everything in the Database is and its purpose is in the context of the game. It then can construct a virtual world using the Data and Simulate everything based on that Data.

Think of it this way. In a restaurant you look at a menu and order a Dish to eat(Your Mech and Match). you don't order all the ingredients! You order the dish. The Kitchen (Dedicated Server) needs to know what all the ingredients are to make the Dish for it to taste good. The Kitchen brings all the right ingredients together and you get to enjoy your Meal (your Mech in the match)

BRAVO BRAVO ) Thank you! Delicious explanation!

[ShooteyMcShooterson]

Are Clan LB20x supposed to have no ghost heat?

Was messing with a Moonwalker build using 3 of them, compared to 3 Clan ac20s, which do get ghost heat. The ac20s wind up being dramatically hotter because of it.

[w0qj]

No ghost heat for LB20x (neither IS nor Clan). But it does give off heat by itself...
https://mwo.nav-alph...pment/ghostheat
https://mwo.nav-alph...uipment/weapons

Edit: If you're gonna fiddle with LBX + Moonwalker, try this:
MCII-MWK(LGD) or MCII-MWK[H]:
2xUAC10 + 2xLB10x + XL325

ShooteyMcShooterson, on 02 December 2023 - 09:41 PM, said:

Are Clan LB20x supposed to have no ghost heat?

Was messing with a Moonwalker build using 3 of them, compared to 3 Clan ac20s, which do get ghost heat. The ac20s wind up being dramatically hotter because of it.

Edited by w0qj, 03 December 2023 - 12:23 AM.

[Besh]

Matt Newman, on 02 December 2023 - 03:17 PM, said:

It does make sense when you consider everything you bring into the match is Data. and the Server needs to know what that Data represents.

Mechs have Item IDs
Armor Has an Item IDs
Infrastructure has an Item IDs
Engine have Item IDs
Weapons have Item IDs
Ammo Has Item IDs
Decals Have Item IDs
Cockpit Items Have Item IDs

Everything in the game has an ID and every account has Mech Loadouts with IDs that contain the IDs of all the Items that make up that Mech.

So we send the Player ID and Mech ID and all the loadout data which includes everything that is attached to that mech which are all Item IDs. Send that all to a Server which has to understand that Data Structure and properties and behaviors associated with those Item IDs.

The server needs to know what everything in the Database is and its purpose is in the context of the game. It then can construct a virtual world using the Data and Simulate everything based on that Data.

Think of it this way. In a restaurant you look at a menu and order a Dish to eat(Your Mech and Match). you don't order all the ingredients! You order the dish. The Kitchen (Dedicated Server) needs to know what all the ingredients are to make the Dish for it to taste good. The Kitchen brings all the right ingredients together and you get to enjoy your Meal (your Mech in the match)

Right now we seem to have exceeded the total number of possible ingredients in our kitchen, the Cooks can't prepare the meal in time due to the overhead of loading the ingredient list to build the meal correctly. Even if you come in to order a burger its still loading the full list of possible ingredients to assemble the burger correctly.

I guess the question then becomes why doesn't the Server have the Data all ready?
Why load all the Item IDS and Data at runtime rather than have that Data present within the server executable?
I don't know the answer to that but I would guess that there is a good reason. Executable size maybe?(I'll check and see if I can find out)

I was/am just not understanding why Server needs to load entire Database at Lobby/Matchcreation . I would have thought every 'Mech/Loadout, including boltons etc. is stored as is, and when used ( sent into Match ), Server just needs to pull those IDs that are being used by the 24 'Mechs in that particular Match from the Database .

To stay in metaphor ( I was working in kitchens ) : When an order for somethng comes in, I don't ned to recall ALL the ingredients we have stored . All I need to know to prepare that meal/dish are the specific ingredients for whats has been ordered ( bun, patty, sauces/drsseing, garnish etc .) . When a veggíe burger is ordered, I dont need to know I could make it with either a ToFu/Legumes/Mushroom Patty . I read its ordered with ToFu patty, and I grab exactly that from storage and use it.

Additionally, having exceeded the max# of Items in context to me would mean either
buffer overflow
or
database overflow .

And this lead me to thinking what Eiji mentioned :

Eiji Sixfinger, on 02 December 2023 - 08:02 PM, said:

I'm confused as to if it's the total number of items in the database, not what we actual are using/own, why is it so intermittent?

~

The intermittence of the Issue to me just seemed to point to a Memory allocation Issue rather than a Database Issue . Or maybe even a HW problem XD .

Anyhow . Thanks . Transparency, time spent to check this Thread and answer individual questions - esp. at weekend! - are much appreciated .

p.s.: Some humblebrag to explain why I am interested in understanding the Problem . I was employed at H&P as Support Tech Networking/Servers & Storage in one of their international Centers a few decades ago . While I haven't worked in the field since, and ofc my "knowledge" ( if it can even be called that ) is more than rusty, asking myself "Whats happening here ?" when looking at a Tech problem still is some kind of autoroutine .

Edited by Besh, 02 December 2023 - 11:17 PM.

[martian]

Besh, on 02 December 2023 - 10:53 AM, said:

This is the part that does not make any sense .

The problem is that some identical junk bolt-on is listed many times in the database of all game assets that must be loaded server-side before every single game.

When loading the new game, the server needs to know what bolt-on it must load and if it can load it on a particular 'Mech.

Strelok7, on 02 December 2023 - 08:46 PM, said:

Are they deleting some bolt-ons, did I understand correctly?

They are not deleting bolt-ons, but they are lowering the number of IDs for one identical bolt-on.

Let us just consider one bolt-on "Wrecked car". So just one example:

As it is now, this bolt-on has many IDs:

• ID1 if it is on AS7-D
  
• ID2 if it is on AS7-D-DC
  
• ID3 if it is on AS7-RS
  
• ID4 if it is on AS7-K
  
• ID5 if it is on AS7-S
  
• ID6 if it is on AS7-K3
  
• ID7 if it is on AS7-KR
  
• ID8 ....
  
• ID9 ...
  
• ID150 if it is on ANH-1A

The same "Wrecked Car" bolt-on can also be used on many other 'Mechs, also with many IDs. Thus, you have the situation when you have created some huge database with hundreds of entries for different IDs for what is essentially one identical bolt-on. And if you add all bolt-ons together, you have a database with many thousands of entries, with bolt-on IDs being a significant part of it.

What is Matt Newman doing? He is changing the system of bolt-on IDs to reduce the size of the database that must be loaded, to return the database into the manageable size. After the December game patch, the server will know that all "Wrecked Car" bolt-ons have ID1 and can be loaded on all variants of the Atlas chassis. Ditto for other 'Mechs.

The database will be smaller and the server will not crash when loading it.

[Besh]

But why does it load the entire Database at match/lobbycreation ? Why does it not just pull whats needed for the specific 24 'Mechs/loadouts/BoltOns etc. neeed for that specific Match ?

Edited by Besh, 02 December 2023 - 11:16 PM.

[martian]

Besh, on 02 December 2023 - 11:15 PM, said:

But why does it load the entire Database at match/lobbycreation ? Why does it not just pull whats needed for the specific 24 'Mechs/loadouts/BoltOns etc. neeed for that specific Match ?

You can ask the PGI employees who created the system. Oh sorry, you cannot - they left MWO mostly in 2015 or so.

[Tarl Cabot]

Besh, on 02 December 2023 - 01:50 PM, said:

Well, forgive me for thinking to be able to understand stuff . I realize its not everyone's cup of tea, but I kindof like doing it .

Here is a question : How is it anyhow feasable for the "the Server" to need to load "the entire item database" for every match to be able to "crossreference Items" ?

Matt hit it on the head, mostly Even Everquest for a number of years had a cap/max numbers, items.. hmm, I believe it was characters, that could be added to a database, going over would exceed server ability to render the game, making the servers unstable.

I am sure someone will correct me, but each drop is an server instance, a VM?, it has to load that database for each game, the pulls/unpacks said data for the game itself, overwhelming the resources of that server instance. If the issue was simply loading that large database which was then being searched/info pulled/unpacked that was causing the instability, most games would not launch successfully.

Most assumed that the no server box was due to trying to drop on the EU server they have not said what else was causing the the instability with EU is yet, But people were still getting the no server (instance) with EU unchecked. Remember, when we drop into a game, we are constantly connected to the authentication server, then said details for the drop are sent to the appropriate game server. So the no server found may have been those instances where the database was having to pull the most info, cockpit items, bolt-on, colors, etc. And I wonder if that also includes patterns and a few other items.

I am sure someone will come along and make corrections to my post. Most of it is from paying attention (and reading way too much) when devs in other games make posts of why something happened or why they cannot do something due to there being a limitation.

Besh, on 02 December 2023 - 11:15 PM, said:

But why does it load the entire Database at match/lobbycreation ? Why does it not just pull whats needed for the specific 24 'Mechs/loadouts/BoltOns etc. neeed for that specific Match ?

It has to pull/load that info for each instance, else the main servers would choke. And based on the information provided, the primary issue is the a combination of the database size + unpacking that data when called, the resources of that instance are exceeded. Else all drops would be kicking everyone back to the mechbay.

Edited by Tarl Cabot, 02 December 2023 - 11:51 PM.

[Matt Newman]

Besh, on 02 December 2023 - 11:15 PM, said:

But why does it load the entire Database at match/lobbycreation ? Why does it not just pull whats needed for the specific 24 'Mechs/loadouts/BoltOns etc. neeed for that specific Match ?

I think that could be an option. however I don't believe that is how the system currently works. With Live operations games that start small and grow over time decisions that once were logical may not be in 12 years time. I am just thankful that we have a non essential feature that we can easily optimize to get our selves back on track with plenty more room to work.

martian, on 02 December 2023 - 11:21 PM, said:

You can ask the PGI employees who created the system. Oh sorry, you cannot - they left MWO mostly in 2015 or so.

Actually a significant number of talented engineers who worked on MWO are still with the company. I know because I have worked with them on MWO since inception and continue to rely on them when we manage to find ourselves in situations like these

[Matt Newman]

AmbidXtrousGNOME, on 02 December 2023 - 08:14 PM, said:

Hey Matt,

does PGI have the means to snap-shot servers and restore a server to a date-in-time via image? I'm speaking from a networking understanding of how I do server backup/restores at my job so I'm going out on a limb and assuming the same could be done for gaming servers since (assuming again) there really there isn't much of a difference from a game server and a high traffic webserver.

Maybe that's what you mean by "rolling back" the November patch or maybe it's an actual uninstall process instead of a restore-state kind of thing. Just and idea.

Thanks for the openness about the issues instead of just giving the classic "It's being worked on" kind of BS.

We do have database snap shots for sure and source control (perforce) on all our game assets and code.
Rolling back is not impossible but it is not always the best option.

When we first started seeing issues I brought all the lead engineers, Database Admin, QA lead, Web Dev, Infrastructure Network/IT and build engineers together and relayed the situation and the player experience. From there they each set about investigating the behaviors of their respective domains (looking at server logs, checking network routing Etc) so we can find the cause and hopefully the solution as soon as possible. After some time of investigating and cross checking with each other we discuss our findings and our options to fix the problem. In this case the reduction of the overall number of items in the game was what we determined would be the best way forward. It was an easy call for it to be bolt-ons as they made up more than 55% of the item IDs, are cosmetic, and had a clear path to optimize.

Rolling back would have potentially taken longer, carried more risk, and been more work. AND we would still be in the same position where we couldn't add more items to the game without doing the work to remove the bolt-ons.

Hopefully that gives you some insight on that.

[martian]

Matt Newman, on 02 December 2023 - 11:45 PM, said:

Actually a significant number of talented engineers who worked on MWO are still with the company. I know because I have worked with them on MWO since inception and continue to rely on them when we manage to find ourselves in situations like these

Maybe with the company, but not working on MWO - at least that is how I understood the Russ Bullock's words when he described the current MWO status. Do I get it right?

[Tarl Cabot]

Matt,
Back tracking on my previous post, thus asking for clarification. Thus a question on the no server found. Is just that player affected or all 24 players for that actual server instance? Just from the lack of PSR movement, it would appear to be that server instance.

Edited by Tarl Cabot, 03 December 2023 - 12:13 AM.

[Besh]

Tarl Cabot, on 02 December 2023 - 11:43 PM, said:

It has to pull/load that info for each instance, else the main servers would choke. And based on the information provided, the primary issue is the a combination of the database size + unpacking that data when called, the resources of that instance are exceeded. Else all drops would be kicking everyone back to the mechbay.

Preempting "I have no clue how the engine works" ( which is true actually ), I would think it would be more efficient re macthinstance to store Individual player loadouts as a hash under that player's Profile . Would just need to be verified as valid re Item IDs at "save" . Then when [Mech] gets sent into match, the instance just pulls the IDs needed for that specific Match ( instead of loading the entire Item Database into each match instance ) .

But *heh*, what do I know ? I am just trying to understand whats going on .

Edited by Besh, 03 December 2023 - 01:26 AM.

[Matt Newman]

Besh, on 02 December 2023 - 10:13 PM, said:

I was/am just not understanding why Server needs to load entire Database at Lobby/Matchcreation . I would have thought every 'Mech/Loadout, including boltons etc. is stored as is, and when used ( sent into Match ), Server just needs to pull those IDs that are being used by the 24 'Mechs in that particular Match from the Database .

To stay in metaphor ( I was working in kitchens ) : When an order for somethng comes in, I don't ned to recall ALL the ingredients we have stored . All I need to know to prepare that meal/dish are the specific ingredients for whats has been ordered ( bun, patty, sauces/drsseing, garnish etc .) . When a veggíe burger is ordered, I dont need to know I could make it with either a ToFu/Legumes/Mushroom Patty . I read its ordered with ToFu patty, and I grab exactly that from storage and use it.

Additionally, having exceeded the max# of Items in context to me would mean either
buffer overflow
or
database overflow .

And this lead me to thinking what Eiji mentioned :



The intermittence of the Issue to me just seemed to point to a Memory allocation Issue rather than a Database Issue . Or maybe even a HW problem XD .

Anyhow . Thanks . Transparency, time spent to check this Thread and answer individual questions - esp. at weekend! - are much appreciated .

p.s.: Some humblebrag to explain why I am interested in understanding the Problem . I was employed at H&P as Support Tech Networking/Servers & Storage in one of their international Centers a few decades ago . While I haven't worked in the field since, and ofc my "knowledge" ( if it can even be called that ) is more than rusty, asking myself "Whats happening here ?" when looking at a Tech problem still is some kind of autoroutine .

Those are fair questions and I have to admit I am not an expert! The presenting behaviors in the software do feel like memory overruns but I am assured by people way smarter than I am that it is the overall number of items in the database that is causing these issues. The Intermittence AFAIK can be chalked up to large data transmission?... lets pretend our database is a Tweet on twitter and we have been keeping every tweet under the Character limit of 300 but somehow we just start tweeting at 310 characters and Elon Musk is mad at us!? (I am getting further out of my knowledge comfort zone must switch back to food based metaphor!!)

Also I am glad you like my Kitchen Metaphor I will say that yes as humans we don't have to load every ingredient when we are making a specific dish! BUT think about how many dishes you know how to make! Or how you know what spices and flavors to add to get a specific flavor its almost like you DO have a Database in there right?