System.gc() in 1.4.2_04

General Discussions
#1

Has anyone being seeing strange behaviour(s) with the GC in the latest Sun JVM?

I had major problems with the JVM’s ultra-greediness on memory when doing some NIO stuff (c.f. my post in networking some time back). Eventually, I moved all the other servers and daemons on to separate machines, so that the server with the JVM is not hosting anything else (which helps to prevent it from crashing linux!) although it does do X11.

I now have a nice simple situation where the JVM does approx 10 streamed parallel request/responses from a remote client, totalling around 50k of data, and takes 295Mb mem to do it. That eats heavily into swap space, so it takes minutes to recover, but still has the almost 300Mb mem usage…

…and yet a System.gc() at any time afterwards instantly (less than 2 seconds) recovers 170+ Mb of that. What’s bizarre is that the JVM apparently happily takes so much memory that it turns a sub 1-second set of processing and turns it into a 2-minute processing (because of the huge delays for huge amounts of mem-swapping!) but doesn’t feel any need to reclaim the memory.

Have we got hold of a new GC recently and I just didn’t notice? Does anyone know of a way to tell it not to be such a greedy **** and that virtual memory is not “there for the taking”, it’s “to be used sparingly” (I’d turn off swap entirely if it were just up to me ;)).

Incidentally, even post-GC it’s still using 128Mb, which is almost twice what it was typically using before changes to the NIO code and the upgrade from 1.4.2 to 1.4.2_04…

EDIT: although we get better performance in the 1.5 beta (uses 100Mb less memory for exactly the same test case) we haven’t yet regressed a version, so haven’t been able to rule-out a JVM-regression bug

#2

What VM options are you using? I read that 1.5 defaults to use -XX:+UseConcMarkSweepGC for he collector if -Xincgc is specified, whereas a different collector was used in that case under 1.4

You might try playing with -XX options to force a particular collector.

#3

Not using any options at all. Sigh. All very confusing.

“My JVM got memory-addiction, and I don’t know why; it’s so bad, it’s even lieing to itself about it’s usage. I’ve tried sending it to Memoryholics Anonymous (profilers), but so far it just wrecks the place each time (machine crash)”.

Now experimenting to see what hardware and OS platforms we can reproduce on…

#4

As far as I know current garbage collectors do not take memory load (e.g. excess swapping) into account. However the only things they could do would be

[] Vary collection frequency
[] Change policy of when the committed heap size is decreased (i.e. how quickly is surplus heap returned to the operating system.
[*] Adjust reclamation of SoftReferences

Currently it appears that SoftReferences are reclaimed before it will consider acquiring more memory from the OS, but once acquired it will tend to keep the memory unless the total of hard and soft reachable memory is sufficient to allow reclamation. In times of memory stress it would be better if SoftReferences were reclaimed more frequently.
The garbage collector doesn’t appear to track the amount of memory allocated as direct buffers or memory mapped files, but again it could only reclaim this if they were merely soft reachable.

#5

Dunno if it was related but I had a similar problem with my NIO network usage a while ago. The memory usage would simply explode even though only a few sockets were active (a loopback connection) and little data was transmitted. It turned out that the problem was the non-direct buffers I used for socket buffers. When using non-direct buffers, NIO automagically creates temporary direct buffers to copy from native to your nondirect buffer. Each direct buffer takes at least 4k each, and they’re not easily freed (have a finalizer). To make a long story shorter, switching all the buffers I was passing to NIO channels from nondirect to direct fixed the problem completely, as it seems the channel could use them directly.

  • elias
#6

[quote]Dunno if it was related but I had a similar problem with my NIO network usage a while ago.

…basically, it looks like the same problem, however I had to make an additional change. Could you please reply (if at all) in the other thread (which is all about the networking + NIO aspects of this problem, as opposed to this where I was just asking about the GC + mem usage aspects)?