Is there a workaround for this

Performance Tuning
#1

I want to stream large amounts of data from the hardrive, i’m using the following code

FileInputStream file_input = new FileInputStream(DTConstants.OCTREE_TEMP_LOCATION+fileName);
FileChannel file_input_chan = file_input.getChannel();
MappedByteBuffer mappedBuffer = file_input_chan.map(FileChannel.MapMode.READ_ONLY, 0, file_input_chan.size());

But after a short amount of time i get this exception
“Not enough storage is available to process this command”

i can circumvent this by setting this:

mappedBuffer = null;
file_input_chan = null;
file_input = null;
System.gc();

But calling the garbage collector is really time expensive, so is there another way of doing this

tia

Paul

#2

why not create one bytebuffer and stream your data through that?

the behaviour you describe is logical… from the documentation:

[quote]A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected.
[/quote]
So, create a direct bytebuffer and read the data through filechannel.read

#3

The problem is when the tiny objects on the heap (refering to large mapped native data) are not garbage-collected. The problem gets worse when the tiny objects make their way out of eden-space to the next heap, which will not be GCed that often.

This is a design-flaw of MappedByteBuffers, which make them unusable for most performance tasks.

#4

Yeah, there should be an unmap() or dispose() call that can be used to invalidate the MappedByteBuffer. Is there a bug # for this ?

#5

So what’s the second best thing to fix this, coz sometimes i need to read in a million points, but othertimes when a node is further away only saw 1000

#6

The best fix is just to use normal File IO and don’t use the Mapped stuff until the RFE for the dispose() or unmap() request is agreed and implemented.

Cas :slight_smile:

#7

public static final void clean(final Object buffer) throws Exception {
AccessController.doPrivileged(new PrivilegedAction()
{
public Object run()
{
try
{
Method getCleanerMethod = buffer.getClass().getMethod(“cleaner”, new Class[0]);
getCleanerMethod.setAccessible(true);
sun.misc.Cleaner cleaner = (sun.misc.Cleaner) getCleanerMethod.invoke(buffer, new Object[0]);
cleaner.clean();
} catch (Exception e)
{

				//e.printStackTrace(null);
				e.printStackTrace();
			}
			return null;
		}
	});
}

this seems to work, but only when i set my mapped byte buffer to ‘null’ ( i get an nullpointer exception ) but speeds seems to be fine.
but when i remove the e.printstactrace, it get slow again.
But i got a hotspot virtual machine error when i point a mapped buffer in it when it’s not null.

pretty weird