Physical + virtual memory

[Mr. Tinker]

I know that a 32-bit OS can only theoretically address 4Gb of physical RAM, and that Windows-32 can only address 3.2Gb physical, but what is the max physical PLUS virtual ram that Windows-32 can allocate to a program?

[Delirious]

according to my microsoft press book "xp inside out" the minimum size is 1.5 times the amount of physical memory and the max size is 3 times that value.

[uOpt]

I know that a 32-bit OS can only theoretically address 4Gb of physical RAM, and that Windows-32 can only address 3.2Gb physical, but what is the max physical PLUS virtual ram that Windows-32 can allocate to a program?

No, a 32 bit OS can address more than 4 GB if it has working PAE. PAE is "physical address extension" and refers to additional addressing pins on the CPU (for more than 4 Gb physical memory) and how to address them from a 32 bit OS.

However, Windows XP has removed working PAE support in SP2, for reasons that escape me.

When it comes to the virtual memory that a 32 bit program can use, that limit is 2 GB (Windows default), 3 GB (Windows with some crazy switch, FreeBSD and normal Linux) or just less than 4 GB (Linux with the 4GB/4GB split patch, and 32 bit Unix programs running in a 64 bit OS).

Your last question about the amount of physical memory that a 32 bit OS can assign to a 32 bit program is somewhat interpretation-worthy. Since the program is usually limited to 2 or 3 GB virtual memory, assigning more than that opens the question where to logically assign more memory. However, if the program is one of those who have difficulty living in 2 or 3 GB of virtual memory they will implement some kind of swapping of virtual memory mappings or using read/write to access more data. If so, the OS will use the filesystem buffer cache to speed up this mappings changing and you could say that a large part of the filesystem buffer cache (which is always directly associated to physical memory) is exclusively for this program. So you could have any amount of physical memory assigned to a program, more than 4 GB.

[Delirious]

Heres something from microsoft.

http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

[Mr. Tinker]

Great answers, guys. Let's get specific here:
-Windows Server 2003
-3D Studio Max v7
-8 render servers (Dell, dual Xeons)
-3Gb physical each.

Our problem is that in larger scenes with lots of foliage, we get "Out of Memory" errors during renders when the memory usage of 3d studio max grows beyond 3Gb. The shot never finishes.

We can break the shot up into distance clipping planes (say, 0'-500' for one render, then 500'-100' for another) but it's a big hastle and I suspect that it's unnecessary.

Does anyone know how to let 3Ds Max access more than 3Gb for a scene?


Edit: Great link there Delirious. I might find that useful.

[uOpt]

Great answers, guys. Let's get specific here:
-Windows Server 2003
-3D Studio Max v7
-8 render servers (Dell, dual Xeons)
-3Gb physical each.

Our problem is that in larger scenes with lots of foliage, we get "Out of Memory" errors during renders when the memory usage of 3d studio max grows beyond 3Gb. The shot never finishes.

We can break the shot up into distance clipping planes (say, 0'-500' for one render, then 500'-100' for another) but it's a big hastle and I suspect that it's unnecessary.

Does anyone know how to let 3Ds Max access more than 3Gb for a scene?


Obviously a 64 bit version would solve this.

For 32 bit in a 32 bit OS, unless there is a Linux version that you could run in a kernel with the 4GB/4GB split patch there is not much to be done.

Even if there is no 64 bit version of your application, you can try running the 32 bit application on a 64 bit OS. In Linux and FreeBSD that gives you 4 instead of 3 GB of virtual memory. It should be the same for Windows but I don't personally use Windows, so maybe they screwed something up.

[Mr. Tinker]

I know that, but changing OS is not an option. I am not in charge of these things, but I know they would buy more ram if they found it beneficial. I pointed one of the guys to the link Delirious mentioned and that's about the best I can do.

I really wanted a solution that would be like "tweak this reg key and your app can now access 3Gb physical + 61Gb virtual."

But I do sincerely thank you folks.

[Jarrod1937]

holy crap... you guy's must be making some very complex scenes.... you could do what i'm doing, building a cluster that will distribute the resources needed out to 3 seperate computers and my main machine.
however i've never gotten that error before with my page file set correctly (i also use 3ds max 7). i have 1 gig of physical ram and i have 2 gigs of page file seperated between my two hard drives (1 drive gets 1024mb max and min and the other gets 1024 mb max and min). however, whats strange is that i've seen 3ds max (by pressing ctrl+alt+delete) i see that it soemtimes uses over 2 gigs... so i'm not too sure.
care to show some of your scenes (since you can't render you can just press ctrl+alt+print screen while viewing the viewport and paste it in photoshop), just curious to see how much detal you're trying to pump out. although i model for games most of the time so that may be why i never got that error messege before.

[Delirious]

It was my understanding that since he is using 8 server that they are all clustered. maybe i read into it to much.

Buying more memory can never hurt.

[Jarrod1937]

i actually think he might already have a cluster... well then maybe his manager doesn't distribute out all to the ram of each server...
either way i don't see how he could be getting a virtual memory low error if he has his virtual memory setup correctly on his main machine. i've rendered a 32,000,000 or so poly scene (it was a big nature scene with a bunch of foilage all modeled) before on one machine and never had a problem.

[Mr. Tinker]

The servers aren't clustered, and I'm not sure if 3DS Max's render manager would utilize that ( I really just don't know much about the subject of clustering).

The error it gets is "Out of Memory" and it referrs to physical memory. I don't know why it won't allocate more virtual memory. Could be virtual memory settings, or maybe the max that Windows can allocate to an app is 3.2Gb regardless of whether it's physical or virtual (that's the subject of my original question).

For the record, the scene I'm working on is: 4047 objects, more than 41,000,000 polys, and that count doesn't include the 2227 RPC plants, AND I haven't even brought cars and RPC people into the scene. I expect the poly count to grow beyond 50M. Also, consider the fact that there are MANY lights and most of them use BIG shadow maps, which hog memory (It's not the way I'd put a scene together, and models are very poly inefficient, but it's 3ds max so that's a given).

The guy in charge of the servers thought as I did, that Windows could only allocate 3.2Gb per app.

[Delirious]

Clustering is jut when a bunch of computers spread the load equally between them. (very simplified explanation).

[uOpt]

More physical memory will never do anything about the virtual memory limit of a 32 bit processor, except if you didn't have enough swapfile.

It is not generally possible for a userlevel process to even detect it ran out of physical memory. Physical memory is RAM + swapfile and it is shared across all processes and the kernel. A userlevel process can generall only run out of virtual memory, unless RAM + swapfile < virtual memory limit.

[Mr. Tinker]

Clustering is jut when a bunch of computers spread the load equally between them. (very simplified explanation).
well, I know a liiiiitle more than that.

More physical memory will never do anything about the virtual memory limit of a 32 bit processor, except if you didn't have enough swapfile.

It is not generally possible for a userlevel process to even detect it ran out of physical memory. Physical memory is RAM + swapfile and it is shared across all processes and the kernel. A userlevel process can generall only run out of virtual memory, unless RAM + swapfile < virtual memory limit.
So what do you think is happening? Thanks for you help so far, btw.

[uOpt]

So what do you think is happening? Thanks for you help so far, btw.

You hit virtual memory, unless your sum of RAM + swapspace is too low. RAM + swapspace should be bigger than your biggest program (obviously).

To test for the latter I would temporarily set your swapfile to 6 GB or somesuch and re-test.

[Mr. Tinker]

Will it be able to allocate more than 3Gb to the app? I thought swapfile WAS virtual memory.

[uOpt]

Will it be able to allocate more than 3Gb to the app?

No.

But you need to make sure the OS "feels" that it has enough physical memory (RAM + swap) to back up those 3 GB, and it needs physical memory for itself. To make sure you should have 6 GB or so physical memory.

Another probem could be that the error message is bogus and that the application actually ran out of some of its own buffers and not out of OS-supplied memory. That happens quite a bit and is one of the reasons I don't use closed source software anymore.

[Mr. Tinker]

Try being an animator using open source. Have fun with Blender, much less trying to convince your bosses and peers to use it.

[[XC] moddolicous]

Maybe a stupid suggestion, but try setting the page file to system managed. Its worth a shot.

[uOpt]

Try being an animator using open source. Have fun with Blender, much less trying to convince your bosses and peers to use it.

My suggestion that he need to changes OSes was actually targetted as he needs 64-bit Windoze.

It should (not tested myself) give that last gigabyte of virtual memory from 3-4 GB to 32-bit applications, hence he would gain 1 GB extra space without changing his application.

[Mr. Tinker]

But we only have 3gb physical installed and no chance of switching to Win64.

[uOpt]

But we only have 3gb physical installed


You still don't understand the difference between virtual and physical memory.

Yoru application runs out of the 3 GB virtual memory limit (if the error message is correct and if you have enough swapspace).

Your application can happily allocate and use 3 GB of memory of you only have 256 MB RAM + 4 GB swapspace. Of course performance sucks but the allocation will work.

The 64 bit OS will give a 32 bit applicaton 4 GB virtual memory, no matter how much RAM you have, as long as you have enough swapspace. 3 GB RAM + a few gigs of swap and a 64 bit OS will give you 4 GB allocateable


and no chance of switching to Win64.

Translation: we don't want you do fix your problem.

With that attitude there's nothing you can do.

It is very unlikely that the people making this decision understand virtual and physical memory either :)