13 Dec, 2012, Sharmair wrote in the 2nd comment:
Virtual memory management by the OS? If you are looking at how much RAM is actually in
use, you might not be taking into account the parts of the virtual memory use that it currently
not in active use and swapped out to the paging file. The fact that the use dropped to such
a low value after the MUD was not in active use for a time is probably the largest clue to this
being the cause.
use, you might not be taking into account the parts of the virtual memory use that it currently
not in active use and swapped out to the paging file. The fact that the use dropped to such
a low value after the MUD was not in active use for a time is probably the largest clue to this
being the cause.
07 Jan, 2013, Satharien wrote in the 3rd comment:
Sharmair said:
Virtual memory management by the OS? If you are looking at how much RAM is actually in
use, you might not be taking into account the parts of the virtual memory use that it currently
not in active use and swapped out to the paging file. The fact that the use dropped to such
a low value after the MUD was not in active use for a time is probably the largest clue to this
being the cause.
use, you might not be taking into account the parts of the virtual memory use that it currently
not in active use and swapped out to the paging file. The fact that the use dropped to such
a low value after the MUD was not in active use for a time is probably the largest clue to this
being the cause.
So I took the time and did some research, cause to be honest I wasn't (at the time) 100% sure what you meant by paging file. I also did some more monitoring of the mud while it sat idle with just me on it. Here are my findings:
VM Size is (according to what I was able to find on google) the virtual memory space in use/allocated for the process, which includes the paged data for the process. I assumed based on what you said in your post that this is where the 'missing' memory usage would have gone(Correct me if i'm completely wrong in that assumption). But this is what I'm seeing.
CPU Time: 19:31:50 … Mem Usage: 10,836K … Peak Mem usage: 39,728K … VM Size: 36,520K
After I reboot…
CPU Time: 00:03:45 … Mem Usage: 40,432K … Peak Mem usage: 40,448K … VM Size: 36,916K
Now it takes me forever to try and replicate the Mem Usage dropping down by so much. I've seen it drop to 35mb, 31mb, 17mb and that above(10,836K), and the mud still runs flawlessly. And each time, the VM Size never changes. In fact, nothing about the program changes except the Mem Usage drops by a drastic amount. And it's not predictable. I can't tell you when or how or why the Mem Usage field is dropping. It doesn't seem to be timed(I've seen it happen at 3 hours uptime and over 1 day of uptime), it doesn't seem to be event based(as i'm not touching the mud, not even the computer it's on other than to move the mouse randomly to look at the task manager). And playing the game after the Mem Usage has dropped, moving around, killing mobs, etc, doesn't boost the muds Mem Usage above what's to be expected(it dropping slightly when a mob dies, it raising again when it repops, etc. Expected behaviour).
Also, regarding the 'not in active use for a time' comment that you made, i'm not sure what you mean by that. The mud is constantly running, it doesn't 'pause'. Mobs are walking around, scripts are being executed, health is checked and updated constantly. Whether someone is on or not. (I have a system in place to activate/deactivate areas to cut down on processing time, since there's no point running scripts/etc in an area no one has been in in 10+ minutes. But that system is currently disabled atm, to keep the mud as active as possible to test this.)
Anyone have any idea why the Mem Usage field is dropping randomly? Or am I completely confused by all of this? I'm open to ideas/corrections where i'm (obviously) wrong :P Something's happening, and I have no idea what it is, lol.
07 Jan, 2013, quixadhal wrote in the 4th comment:
What is meant by "not in active use" is that modern OS's don't swap, they page.
Back in the Olde Dayze of Yore, processes used to be fully swapped in and out of memory as they ran…. so when your application got its turn to run, the entire process (code + data) was pulled in from disk, ran for some amount of time, and then could be swapped back out to disk if something else needed memory.
Nowadays, your application is paged in and out, as needed. Code or data pages which are not accessed are paged out to disk and not reloaded unless something needs them.
So, let's look at your example. You have a MUD which takes around 40MB of storage, split between code and data. Odds are, there are LOTS of chunks of code which are never called, or only very occasionally called, and those gradually get paged out to disk. Likewise, if you have data elements associated with those seldom-used bits of code, they will also get paged out.
On a typical linux system, the page size is 4K, so if you have (for example) a bunch of magic spells, odds are pretty good that without a large and busy player base calling those routines, many of them will get paged out. Along with them, large string descriptions of rooms that nobody ever visits will get paged out.
When you reboot your game, it has just loaded everything and so nothing has, yet, been deemed idle by the paging system, hence it will sit at full memory use for a while.
This is why many of those coding techniques which seemed so clever back in 1990, like shared strings and unloading areas with no players in it, are kindof a waste of time these days. The OS does almost as good a job, and it's far less likely to be buggy than your own code. :)
Back in the Olde Dayze of Yore, processes used to be fully swapped in and out of memory as they ran…. so when your application got its turn to run, the entire process (code + data) was pulled in from disk, ran for some amount of time, and then could be swapped back out to disk if something else needed memory.
Nowadays, your application is paged in and out, as needed. Code or data pages which are not accessed are paged out to disk and not reloaded unless something needs them.
So, let's look at your example. You have a MUD which takes around 40MB of storage, split between code and data. Odds are, there are LOTS of chunks of code which are never called, or only very occasionally called, and those gradually get paged out to disk. Likewise, if you have data elements associated with those seldom-used bits of code, they will also get paged out.
On a typical linux system, the page size is 4K, so if you have (for example) a bunch of magic spells, odds are pretty good that without a large and busy player base calling those routines, many of them will get paged out. Along with them, large string descriptions of rooms that nobody ever visits will get paged out.
When you reboot your game, it has just loaded everything and so nothing has, yet, been deemed idle by the paging system, hence it will sit at full memory use for a while.
This is why many of those coding techniques which seemed so clever back in 1990, like shared strings and unloading areas with no players in it, are kindof a waste of time these days. The OS does almost as good a job, and it's far less likely to be buggy than your own code. :)
07 Jan, 2013, Satharien wrote in the 5th comment:
Thank you for the quick and clear response Quixadhal, it makes more sense now. The only question I have now is, where is this paged data visible? I was under the assumption that the VM Size was where the paged data was being held(at least from what i've been able to find on google). Is that correct? If so, why does it not change(increase) when stuff gets paged(Mem Usage decrease).
Once again, thanks for the awesome response(and so quickly!). It now makes sense, except for the question above.
Once again, thanks for the awesome response(and so quickly!). It now makes sense, except for the question above.
07 Jan, 2013, Tyche wrote in the 6th comment:
It's possible to see large differences in working set size with MySQL on Windows.
If you've a large result set and you close a cursor, I would expect to see a large drop.
Try the following utility:
http://technet.microsoft.com/en-us/sysin...
You might be able to drill down to get more information.
If you've a large result set and you close a cursor, I would expect to see a large drop.
Try the following utility:
http://technet.microsoft.com/en-us/sysin...
You might be able to drill down to get more information.
Random Picks
Details:
I will reboot the mud now, 3 times, to show you what I mean.
Start: 52,339kb
2nd: 40,532kb
3rd: 50,316kb
The above examples are the 'same' each time. Meaning no code changes, no additional mobs spawning or anything like that(I'm the only player on for all examples).
Additionally, I left the game running idle (no one logged on or off) for 3 hours, when I came back to the PC the game was running at 1,994kb! I thought it had crashed given the change in size, but I was able to walk around, buy and sell items, fight mobs, cast spells, everything. It was running smooth as butter. Then I decided to reboot, and back up to 50mb range, and was still in that range 10 hours later. Anyone know why this would have happened?
This is all pretty confusing to me… I had taken a break from coding(nearly a year with only minor tweaks here and there), but shouldn't code be … predictable to some degree? If nothing is changing, shouldn't it be the same size in memory every time? Also, how the f… did it get down to 1,994kb after 3 hours of sitting idle but yet after 10 hours (trying to replicate that) it's still at 50mb.
Anyone have any ideas of what could be causing this?
Edit: Language is C++ using Lua as a scripting language and MySQL as a database, running on windows XP.