Memory Allocation & Game Crash at 1GB RAM usage

[Chris10]

Iam sorry to be so adamant about this but Iam not willing to give up that easily on this highly important issue which has already been object of
various discussions in the past and during the last few days in this topic for example
viewtopic.php?f=148&t=35669

Subject of this essay is the
The memory usage of PzC and the 1GB crash !

Let me say somehting up front....If I may sound overly critical at some stage I beg you pardon as english is not my native language...
and now lets move to the important things

1. Wherever we acknowledge it or not the game is a memory hog for no good reason...it builds up memory and depending on how many units on the map and how many different animations/sounds this goes very quickly...freeing routines for sounds/animations after each player turn could help a lot but have been discarded...not directly but indirectly...lot of coding with maybe unpredictable results ?...maybe to much of a hassle since the game is nearly build ?...its a mystery
2. This high memory usage may not be an issue for the relative small campaing maps and poeple rarely play so long to encounter the issue that the application reaches 1 GB and goes bust but this cant be really an argument for not doin the right thing and commercialize an engine with an artificial muzzle brake
3. The game crashes when reaching 1GB of RAM usage (evidence below)
4. The mentioned issues are surely dooming a series of current and possible future user made scenarios,campaigns and/or mods to death <VPaulus sound mod> Bigger Maps combined with Massis New Tile Sets > and so on...most probably including mine too where I have already spent over 100 hours only in the map..again..for no good reasonan cause there are possibilities

Now let me be clear..I fully understang that the game shall be playable on systems who only have a total of 2GB system RAM. However, since RAM is cheap nowadays and the bigger chunk of people who play games have at least 4GB RAM on Win7-32bit systems by now I dont see why we all have to suffer the consecuences only to enable a dwindling minority to play the game on their obsolete machines while all others can not only not take advantage of their systems but are even heavily penalized by the current setting not mentioning the mod comunity which is highly restricted this way.

I ask VPaulus to run a test with PzC 1.5 vanilla and his superduper crash scenario for demonstration purposes.
This scneario contains over 100 different units and uses his sound mod..when moving,attacking with most units the game quickly reaches close to 1GB RAM and 100% positive crashed then....which is no wonder since its amended to run on 2GB system RAM machines and wouldnt allow anyway for using more RAM,would it ?...specified highest user adress ?.....


Anyway..I flagged the header of the exe to Large Adress Aware and VPaulus gave it another run on his Win7X64 with his mega crasher scneario and voila we have a blockbuster...for the first time ever...the scenario did not crash cause it now adresses more memory...


- In Win XP Prof with 3GB switch the LAA exe lasted to 1,4GB then got runtime error (I guess faulty memory space reservation or bad garbage collection)...>freeing routines ???....still...record.
- Win7 32bit it crashed at 1,5GB
- Win7-64x with the LAA.exe the game lasted with 1.5GB load until Paulus gave up as he couldnt stress the application more...
...all proove that RAM beyond 1GB is not only pipe dream with PzC

I politely ask now for a definite official solution from the Developers side so that all poeple with at least 4GB RAM Win7- 32bit or more under X64, can adress more memory, including those with Win XP professional with the 3GB switch enabled...
The solution could be:
1. An option in the launcher where people choose 2GB system RAM or Higer and then a helper DLL Sets or Unsets the LARGEADDRESSAWARE bit on the PzC exe > changes the file header accordingly or specifes another highest user adress (preferable !?)
2. or directly 2 different exes
3. since I know the LAA fix works on some systems and on others not any other method you find convenient as long as it has the same result > more memory usage and no crash/freeze on 1GB RAM on systems with more RAM
maybe a PzC.exe which can properly adress the max 1.75 GB for a 32bit application seems the perfect solution apart from the regular PzC exe for the 2GigaBiters but this is obviously not my call...

I think that is a fair request and does not even requiere great effort but only a bit of discernment and sympathy...

Thank you for your consideration Gentlemen
Best Regards
Chris Klein

EDIT

After further testing VPaulus were able to stress the RAM load to 1,8GB on Win7-X64 with an absurd amount of different units and different sounds when it finally crashed..So a 1,75GB solution for all those with at least 3GB usable system RAM seems the best approach as it is doubtful that under real circumstances such a high load is reached

[Chris10]

Game data is not stored in the stack, only local variables are, so there is no chance we could exceed a 1GB stack. It is the heap which causes the problem, and both virtual and physical memory could be an issue. On 4GB and up systems only virtual space can be a problem, but the issue will only happen after the game has been running for a long time.

Unfortunately on bigger scenarios with lots of different units the issue can happen within 1-2 turns and as per now it seems PzC exe does not adress any more RAM than aprox. 1 GB anyway... Otherwise it seems unlogical that it crashes around 1GB on machines with +2GB RAM either x86 or x64
I dunno..this is strange

[lordzimoa]

As far as I know, we don`t have official scenarios that even come close being too large, in map size and units, to cause such a memory crash?

Or do I misunderstand and are there scenarios now in AK causing this?

[Chris10]

As far as I know, we don`t have official scenarios that even come close being too large, in map size and units, to cause such a memory crash?

Or do I misunderstand and are there scenarios now in AK causing this?

No..but a lot of user mods reach and will/would need beyond 1GB of RAM usage but can not be played since the game crashes around 1GB..except if the exe is amended to adress more RAM for systems with 2GB+ system RAM...
May I kindly ask to reread the entire essay
Iam sorry if I elaborated badly..any questions on the matter are happily answered

[Rudankort]

Unfortunately on bigger scenarios with lots of different units the issue can happen within 1-2 turns and as per now it seems PzC exe does not adress any more RAM than aprox. 1 GB anyway... Otherwise it seems unlogical that it crashes around 1GB on machines with +2GB RAM either x86 or x64
I dunno..this is strange

PzC is a 32-bit application. As such, it is able to address 4GB of virtual address space. It does not matter how much physical memory is available, 4GB of virtual memory is limit. From these 4 GB, only 2GB are available to the application - the other 2GB is used by the system.

2GB available to the application, but alas, many regions of this space are used without committing physical memory to them. For example, if I indeed used a 1GB stack, 1GB of address space would be lost instantly, even though physical memory used for the stack would be just a few KB. Fortunately, the largest stack I use is 64KB. Still, because of reserved areas, and also because of fragmentation, the app can use less than 2GB of physical memory. Exact number depends on a lot of factors - version of operating system, installed drivers etc. For example, on my system PzC can successfully allocate and use 1.3GB of physical memory. This is not something special for PzC either - thus, 32-bit Firefox stops working normally when reaching the same threshold too.

The only real way to solve the problem is to reduce PzC memory footprint.

[Chris10]

PzC is a 32-bit application. As such, it is able to address 4GB of virtual address space. It does not matter how much physical memory is available, 4GB of virtual memory is limit. From these 4 GB, only 2GB are available to the application - the other 2GB is used by the system.

2GB available to the application, but alas, many regions of this space are used without committing physical memory to them. For example, if I indeed used a 1GB stack, 1GB of address space would be lost instantly, even though physical memory used for the stack would be just a few KB. Fortunately, the largest stack I use is 64KB. Still, because of reserved areas, and also because of fragmentation, the app can use less than 2GB of physical memory. Exact number depends on a lot of factors - version of operating system, installed drivers etc. For example, on my system PzC can successfully allocate and use 1.3GB of physical memory. This is not something special for PzC either - thus, 32-bit Firefox stops working normally when reaching the same threshold too.

The only real way to solve the problem is to reduce PzC memory footprint.

I agree with what you say..however it doesnt really explain why PzC crashes for so many people around 1GB but with the large adress aware flagged exe goes up to 1,5 GB on Win7-x86 and 1,8 GB on Win7-x64 where then I guess a fragmentation error causes a crash

The only real way to solve the problem is to reduce PzC memory footprint.

You are right of course...but given the simplicity of the engine the only way I see Is dumping all sounds/animations each turn as these are the memory eaters...most other data is static and does not contribute to the memory build up...


Knowing all that...any precise intentions to do something on the matter which would help big mods/scenarios not to crash mid game cause of memory issues ?

[Rudankort]

however it doesnt really explain why PzC crashes for so many people around 1GB but with the large adress aware flagged exe goes up to 1,5 GB on Win7-x86 and 1,8 GB on Win7-x64 where then I guess a fragmentation error causes a crash

How many people are "so many"? Just curious. As for the large address aware flag, I don't see what's not clear to you here. Program has more address space to use (3GB instead of 2) => it can use more physical memory too (still below the limit, but the limit is now 1.5 times larger).

Knowing all that...any precise intentions to do something on the matter which would help big mods/scenarios not to crash mid game cause of memory issues ?

I'll do my best but no promises.

[Chris10]

however it doesnt really explain why PzC crashes for so many people around 1GB but with the large adress aware flagged exe goes up to 1,5 GB on Win7-x86 and 1,8 GB on Win7-x64 where then I guess a fragmentation error causes a crash

How many people are "so many"? Just curious. As for the large address aware flag, I don't see what's not clear to you here. Program has more address space to use (3GB instead of 2) => it can use more physical memory too (still below the limit, but the limit is now 1.5 times larger).

yeah...sure...I just wonder because the memory was there before too, especially in X-64 with lots of RAM...I mean the application should adress 2GB but crashed at 1GB even though there was lots of system RAM available (physical and virtual) ...while with the LAA is just goes up to 1.8GB

Since until now few people run massive modifications obvioulsy the so "many" dont seem so "many" in total numbers but those using Paulus sound mod with other unit packs and stuff come across crashes around aprox. 1GB

Knowing all that...any precise intentions to do something on the matter which would help big mods/scenarios not to crash mid game cause of memory issues ?

I'll do my best but no promises.

Well..That^ll has to do I guess

[Rudankort]

yeah...sure...I just wonder because the memory was there before too, especially in X-64 with lots of RAM...I mean the application should adress 2GB but crashed at 1GB even though there was lots of system RAM available (physical and virtual) ...while with the LAA is just goes up to 1.8GB

I have already explained, the problem has nothing to do with available physical RAM. Run out of virtual memory -> cannot allocate more memory -> crash. As for crashing at 1GB with 2GB addressable, it can happen due to a lot of reasons, like fragmentation. Keep in mind that animations in particular are very large blocks of continuous memory. It is possible that no large free ranges of address space exists, even though the sum of all remaining small ranges in address space amounts to a gigabyte. LAA could get higher simply because it allocates memory in smaller chunks. But I cannot load a big PNG into a bunch of small memory blocks, I need a single large block for it.

[VPaulus]

How many people are "so many"? Just curious.

I don't really know the extension. I suspect not large numbers, but I'm sure those who use my mod, and play several scenarios without exiting to Windows, will happen more often.

As for the large address aware flag, I don't see what's not clear to you here. Program has more address space to use (3GB instead of 2) => it can use more physical memory too (still below the limit, but the limit is now 1.5 times larger).

Then why not release the PanzerCorps.exe flagged for large addresses? At least those with 4GB of memory could benefit and thus avoid crashing Panzer Corps.
So far the only mod who stressed Panzer Corps was my sound mod. But we have already a mod with a tile package, and in future will have new animations, larger maps, all these combined will probably begin to stress Panzer Corps.
And you know, Alex, Panzer Corps has a life besides the official products. Modding can extend the commercial life of a product, that's a known factor.
But with all your "pedigree", I know that you know that. So in the end I'm confident that you will help us.
Thanks.

[Chris10]

Run out of virtual memory -> cannot allocate more memory -> crash. As for crashing at 1GB with 2GB addressable, it can happen due to a lot of reasons, like fragmentation. Keep in mind that animations in particular are very large blocks of continuous memory. It is possible that no large free ranges of address space exists, even though the sum of all remaining small ranges in address space amounts to a gigabyte. LAA could get higher simply because it allocates memory in smaller chunks. But I cannot load a big PNG into a bunch of small memory blocks, I need a single large block for it.

yeah..ok..that does makes sense...
thank you for pointing that out...
so...having an official LAA exe would be of great help then if the overall memory footprint can not be adressed in the remaining time until release ....

[Rudankort]

Then why not release the PanzerCorps.exe flagged for large addresses? At least those with 4GB of memory could benefit and thus avoid crashing Panzer Corps.
So far the only mod who stressed Panzer Corps was my sound mod. But we have already a mod with a tile package, new animations, larger maps, all these combined will probably begin to stress Panzer Corps.
And you know, Alex, Panzer Corps has a life besides the official products. Modding can extend the commercial life of a product, that's a known factor.
But with all your "pedigree", I know that you know that. So in the end I'm confident that you will help us.
Thanks.

The problem is, using IMAGE_FILE_LARGE_ADDRESS_AWARE flag could have side-effects. Some issues occur even with Microsoft's own APIs (see the discussion here for example). Other issues could occur with third-party code I'm using, in particular with HTMLayout. It is a very complex library, which is a complete black box to me. I have no means to check if it is compatible with extended address space or not. Things are made even worse because the bugs related to IMAGE_FILE_LARGE_ADDRESS_AWARE are volatile. Imagine I create an object in memory and pass its address to a system API. In 90% of cases it might get created in the lower 2GB of address space, and the game will just work. And only in 10% of cases it will be created above those default 2GB and cause a problem. So, the end result of this change could be that the game becomes less stable for people who never experience memory problems in the first place (yes, most players still do not install any mods, and this is unlikely to change). Besides, even if it works, it does not guarantee anything. If we are talking about perspectives here, not just current situation, an ambitious mod can easily exhaust even the additional memory obtained from IMAGE_FILE_LARGE_ADDRESS_AWARE.

It is perfectly possible to distribute the modified EXE unofficially, and you don't even need me for that - the require flag can be changed in the EXE file on your side. But on my side I would prefer to find a cleaner and a better solution. The most realistic idea I see right now is to provide a command-line switch, or a registry setting, or something, which would turn caching off altogether in the game. Which means, all sounds and animations would play from disk. Of course, this would reduce performance, but at least it would cover any possible future mods on any PCs with any amount of memory reliably.

[VPaulus]

It is perfectly possible to distribute the modified EXE unofficially, and you don't even need me for that - the require flag can be changed in the EXE file on your side. But on my side I would prefer to find a cleaner and a better solution.

Yes, you're right. I understand that.

The most realistic idea I see right now is to provide a command-line switch, or a registry setting, or something, which would turn caching off altogether in the game. Which means, all sounds and animations would play from disk. Of course, this would reduce performance, but at least it would cover any possible future mods on any PCs with any amount of memory reliably.

I wouldn't mind to test a solution like that. Maybe it could work well and probably we wouldn't loose so much performance. That could be a compromise.

[Rudankort]

I wouldn't mind to test a solution like that. Maybe it could work well and probably we wouldn't loose so much performance. That could be a compromise.

Great. With Africa release approaching I don't have a lot of time, but if you can help with some testing, I would also love to try a few ideas and test some different approaches to attacking this problem, until one of them (hopefully) works satisfactorily. Let us do some test rounds and see where this will bring us. What's the right email address to send test EXEs to? (You can just drop an email to rudankort@rsdn.ru, so that I can reply to it).

[Chris10]

The problem is, using IMAGE_FILE_LARGE_ADDRESS_AWARE flag could have side-effects. Some issues occur even with Microsoft's own APIs (see the discussion here for example). Other issues could occur with third-party code I'm using, in particular with HTMLayout. It is a very complex library, which is a complete black box to me. I have no means to check if it is compatible with extended address space or not. Things are made even worse because the bugs related to IMAGE_FILE_LARGE_ADDRESS_AWARE are volatile. Imagine I create an object in memory and pass its address to a system API. In 90% of cases it might get created in the lower 2GB of address space, and the game will just work. And only in 10% of cases it will be created above those default 2GB and cause a problem. So, the end result of this change could be that the game becomes less stable for people who never experience memory problems in the first place (yes, most players still do not install any mods, and this is unlikely to change). Besides, even if it works, it does not guarantee anything. If we are talking about perspectives here, not just current situation, an ambitious mod can easily exhaust even the additional memory obtained from IMAGE_FILE_LARGE_ADDRESS_AWARE.

It is perfectly possible to distribute the modified EXE unofficially, and you don't even need me for that - the require flag can be changed in the EXE file on your side. But on my side I would prefer to find a cleaner and a better solution. The most realistic idea I see right now is to provide a command-line switch, or a registry setting, or something, which would turn caching off altogether in the game. Which means, all sounds and animations would play from disk. Of course, this would reduce performance, but at least it would cover any possible future mods on any PCs with any amount of memory reliably.

Thank you very much for you consideration. I understand your concern regarding the LAA exe. I posted this because the LAA exe flag has been set to many exes from Fallout3, New Vegas, Skyrim ,Silent Hunter and many many others by now but some companys threatend people with lawsuits cause they considered changing the exe header as copyright infringement even if it not implicated decompiling and/or reverse engeneering the source code...The Creative Assembly is a good example...So I ask for an official exe to not get into trouble by distributing a modified exe....of course if we would get official approval there is no need to discuss this further but we should get some official ok I guess...maybe Slitherine could host a little downloadlink in the announcements section in the mod forum for the LAA exe (with disclaimer, own risk and so on) so distribution would still come from Slitherine...
...

As for the solution you have in mind...a tiny little performance loss will hardly be noted I think. I like this approach...again,thank you very much
for bothering with this
Naturally I offer help in testing new settings and different approaches to see how RAM load is affected