GL_OUT_OF_MEMORY on glTexImage3D()

OpenGL Development JOGL Development
#1

The title pretty much says it all… I guess my question is whether anyone knows of a specific kind of mistake that could lead to this condition?

At this point I am assuming it to be a bug in my code, because I have confirmed 2 things, that

A) this occurs on 2 separate machines at about the same point (one of which has 1GB of ram, the other of which has 3GB)
and
B) our overall application consumes in general nowhere near 3GB. More typically, well under 1GB.

But so far I can’t find any mention of specific (types of?) mistakes that can lead to this, aside from of course actually being out of memory.

[Edit:

Upon further testing & examination, I can cause this error to crop up much faster by increasing the -Xmx setting for my application.

If it helps, I’m running the following:

System A:
XP Pro sp2
3GB RAM
6800GS AGP
Nvidia driver 84.21 (old, I know; but I get really really bad artifacts under 93.71 and any other newer versions i’ve tried; haven’t gotten around to fixing that yet)

System B:
2K Pro sp4
1GB RAM
6800GS AGP
Nvidia driver 84.21

]

#2

I think the simplest explanation is that you are simply using to many textures and the GPU can’t/doesn’t “support” that much texture memory. Can you give us an idea of how many and what the dimensions are?

Though I’m a little surprised that -Xmx would have any affect on it, I would think that the drivers would live in their own address space not that of the application or the JVM.

I’ve previously loaded up to 2GB worth of 2D textures with out any problem, on a system with 1GB of ram, and 64MB of VRAM and an ATI GPU, so the problem may very well be with Nvidia’s drivers. Of course I now do memory mapping with glTexSubImage*() and have no more than 3-4MB of textures loaded at any given point in time ;D.

#3

As an update, I think we are hitting a non-OpenGL limit, although I am surprised that allocations by the gfx driver on my behalf do seem to come out of my application’s memory space. As best my coworker and I can figure out, it looks like the OS is limiting our application to around 2GB. I’m only loading 2 shaders, a handful of textures, and a few display lists, but one of the textures can get pretty big (as large as ~200MB).

We are still a bit hazy on the details of what’s happening, but it appears that the JVM is grabbing 1GB+ of RAM right from startup, (fairly high heap size settings; big memory-hungry application) and then as our app runs, it consumes more, up to around 1.3GB, then when the texImage() call gets fired off… I appear to be hitting a cap somewhere around 1.5GB, which I’ve seen mentioned in several windows-related web postings, although nobody really seems to have a 100% understanding of it, or a really good workaround for it.

My leading theory at this point is that we’re probably just going to have to cut out some dead weight; trim down this mammoth application, which i’ve been hoping for a chance to do anyways.

#4

The Intel 32-bit architecture (x86) gives a process a 4GB flat address space. However the upper 2GB are reserved by Windows for shared memory, system libraries, and address space mapped to PCI devices. So you’re pretty much stuck with 2GB unless…

…you boot Windows with the /3GB flag (check your c:\boot.ini file, duplicate the existing entry and add the /3GB switch at the end). Note however, for an executable to see 3GB it must be explicitly marked as capable of handling the extended address space (clever folks do pointer arithmetics using ints you see…). This is a linker option in Visual C++ (/LARGEADDRESSAWARE). Obviously, you don’t want to rebuild the JVM from source (maybe you do), however you can alter this flag for java.exe (or javaw.exe, whichever you use to launch your application). The Microsoft tool that does this is called ‘editbin.exe’ - I’m sure it’s part of the latest Windows SDK or available elsewhere. BEFORE YOU GO AHEAD and try this I should point out that /3GB is known to reduce system stability, the reduced address space for OS and PCI devices comes at a cost…

Another alternative is to use Intel PAE (Physical Address Extensions) which is like emm386.exe back in the old days :slight_smile: - although I think that works only on Windows Server and I doubt the Sun Java VM would support it.

Last but not least, have you considered going 64-bit? Even if you don’t want to run a 64-bit JVM, the 32-bit JVM marked as LARGEADDRESSAWARE would see a 3GB address space…

Hope this helps,
Matt.

PS: Maybe Ken or someone else from Sun would like to comment on how well tested the 32-bit JVM is for this scenario, and why it is not LARGEADDRESSAWARE by default…

#5

I’m still curious about the gfx and the application’s memory space. I wonder if it isn’t an “optimization” being done to cut down on the number of memory copies, because I know that the textures don’t get uploaded to the board until they are used, I wonder if the driver tries to play it fast and lose with not copying the data out of the application space and into the driver’s AGP or PCIE space until it is first used?

If it’s easy to do you might want to try loading your largest textures first, then use them to draw something to the screen, this will cause the texture to be uploaded all the way to the board and subsequently the AGP memory as well. If the drivers were doing that optimization I described there would then be a copy in driver space and hopefully the application space copy will be freed.

That’s just a thought and I’m thinking out loud, though if it works let us know!

#6

for such outOfMemoryError which frequently occurs when the heap space is overflowed, I implemented a simple heap cache manager. I found some code at Koders under SoftCache sources topic.
It is the best way to avoid those heap limitation and to enter over-heap-sized memory.
The reason why these errors occur is that every object instanciated in java are stored in the back-heap memory, which is limited by the Xmx argument of the JVM at startup. Thus you can increase this parameter at startup but it is not allowed while the JVM is already running.
Java also specfies a RTSJ (Real-Time Specs Java ) API to enter over-heap-sized memory. But I don’t recommend it while I used my own impl of heap memory management.
::slight_smile: