Is anyone using indirect buffers safely yet?

General Discussions
#1

(in reference to the mega-memory leak in java 1.4.x at least up to 1.4.2_05 which causes use of more than a dozen or so indirect buffers to cause a memory leak of up to 500Mb in one minute)

I haven’t seen any sign that Sun have fixed this yet, but I’m fed up of using direct buffers for everything simply because the indirect ones will destroy the app in minutes. And it’s causing headaches, like the fact that I don’t have a recycling class, so run out of direct memory eventually (due to that other, more famous, bug that sun’s dumbass implementation statically assigns a tiny amount of memory (64Mb off the top of my head…) for direct buffers and will NOT make use of the actual RAM in the machine)

So…I was just wondering…has anyone who’s had problems with indirect buffers in the past found that a more recent JVM from Sun is behaving better?

#2

PS: obviously, I hit this problem usually in networking, but I’m assuming it’s a similar set of problems for people working with graphics hardware?

#3

/me is even more fed up than ever with doing searches on NIO problems and issues and always finding his own posts, articles, and attempts to document it

I mean, honestly, it sometimes feels like I’m the only person in the world who uses NIO in non trivial ways. Sob.

#4

I’ve only ever used direct buffers. I think the 64MB limit on them was a choice they made because there is no way to free a buffer… you must wait for it to be collected. So there could be problems allocating many direct memory buffers because the native allocations don’t trigger garbage collection.

#5

[quote] I’m assuming it’s a similar set of problems for people working with graphics hardware?
[/quote]
I don’t think anyone using NIO for graphics hardware stuff (OpenGL) would use indirect buffers. The main benifit of the buffers is that they can be direct so the pointers can be shared easily with native code and no copying needs to happen. Indirect buffers are totally different. In terms of interacting with native code they are useless compared to direct buffers.

#6

What memory leak?

Cas :slight_smile:

#7

Do you mean when you will use NIO buffer for writing, it will use all of your memory?

#8

[quote]What memory leak?

Cas :slight_smile:
[/quote]
open sufficiently many small buffers (something around 10-20 at once IIRC was enough to trigger it, each no more than 5k in size) and your JVM soon gets killed by the OS because every seond the mem usage goes up by megabytes, without limit, and for no reason at all.

Change the line of code “allocateIndirect” to “allocateDirect” and the memory leak vanishes and you can have hundreds of small direct buffers with no other code changes.

#9

[quote]I’ve only ever used direct buffers. I think the 64MB limit on them was a choice they made because there is no way to free a buffer… you must wait for it to be collected. So there could be problems allocating many direct memory buffers because the native allocations don’t trigger garbage collection.
[/quote]
…which is exactly my problem now. There’s no way to manually force a collect, or to find out when it’s happened (unless you start playing with WeakReferences and getting clever and discovering when things disappear, perhaps?) so even writing a buffer manager isn’t easy (thanks, Sun) - but not having one at all to start off with is even harder.

IIRC the solution that grex uses is an aggresive closing of connections and correspondingly aggressive re-use of buffers for new connections, as a workaround whilst indirect buffers are unusable. But I don’t have access to that, and may well have to roll my own either way :(.

#10

What a strange bug. Have you got a simple test case that exhibits the problem?

Cas :slight_smile:

#11

[quote]What a strange bug. Have you got a simple test case that exhibits the problem?

Cas :slight_smile:
[/quote]
There’s a thread on it somewhere in the netwrorking forum, started around 12 months ago. Given the amount of effort it takes to get NIO to do anything the only “small” test cases I ever had relied on a large amount of proprietary library code. The clincher was discovering that after man-months of reading only a few thousand lines of code no problem was ever found, but that direct BB’s instantly solved it.