performance 'hiccups' in IBM & BEA JVMs

Performance Tuning
#1

Hi All

This isn’t games related, but it is performance related, so, oh well… ;D

One of the services my company provides is a WAP-based chat service. We’re rapidly approaching 1 million page views a day (not there yet, but getting closer every day) just in NZ & Australia alone, with Canada probably going to be added in a few months, and perhaps the US shortly after. While our database & app server boxes aren’t even breaking a sweat at the moment (never much more than 15-20% of CPU utilised even during peak), I’m constantly looking at various ways to improve performance (clustering technologies and so on), in case we suddenly get a huge increase in load.

The last couple of days I’ve been looking at IBM & BEA’s virtual machine offerings, but getting some weird/mixed results. I’ve got a JMeter test which covers the full range of functions in our chat service. On an admittedly crappy test server with 512MB of RAM, using Sun’s VM, the test maxes out at about 85-88 requests (page views) per second for about 28000 requests, then up to between 95-100 per second on the second run of the test.

With Bea’s JRockit, the test hits about 160-170 requests per second up to around 12000 requests, and then the VM has a massive pause (every thread stops) for about 20 seconds (the CPU goes completely idle during this period). It eventually recovers, runs again for a while, then pauses again. The final result is only about 96 requests per second because of those pauses.

IBM’s VM exhibits a similar pattern, although the pauses aren’t as long, nor is the peak request rate as high. Final result is about the same though.

Has anyone else experienced similar effects? Or any thoughts as to the cause? I’ve run JRockit in debug mode and it seems to be running code generation during those pauses, but as to why that would kill the entire VM for 20 seconds, I’ve only got guesses.

Couple of final notes:

  1. on the second run of the test JRockit peaks at around 212 requests per second, but eventually tails off again because of the lengthy pauses.
  2. I’m running all VMs with whatever GC model I think will result in the lowest GC pauses (Sun=ConcMarkSweepGC, JRockit=gencon, IBM=optavgpause).
  3. technology wise (if it lends any clues) we’re running Apache+mod_jk connecting to Jetty (using velocity + half an appserver to produce the content), and the database is postgres.

If anyone has any thoughts, I’d appreciate the input.

Thanks,
J

#2

I would say you need to profile your server’s memory usage patterns. It seems full garbage collection cycles are initiated from time to time. Perhaps using a few pools instead of always allocating a new object will change things.

I would be interested in testing our product, Excelsior JET, against your app. It is a Java VM with AOT compiler and it uses a synthetic GC algorithm with reasonably short pauses. But I won’t be able to do that before July 10 or so, because we are about to release version 3.6 and then we shall present it at JavaOne. In the meantime, you are welcome to download the evaluation package and give it a try.

#3

Pretty much everything that makes sense to be pooled is pooled. And we don’t experience any huge pauses with Sun’s VM hence my question.

I’ll try to download the Linux version of Jet (sorry, should’ve mentioned the platform is Linux) when I get a chance and see how I go. Problem is, there’s so many bits to be compiled (web server, app server, etc) I think I’ll struggle to get it all working successfully.

#4

[quote]I’ll try to download the Linux version of Jet (sorry, should’ve mentioned the platform is Linux) when I get a chance and see how I go. Problem is, there’s so many bits to be compiled (web server, app server, etc) I think I’ll struggle to get it all working successfully.
[/quote]
Well, this sounds very interesting. We are in need of some real-world Java server applications to evaluate the performance differences delivered by our product. Could you possibly provide your application to us (under an NDA?) We would set up the necessary project files and build scripts and email them back to you. No purchase obligations of course.

#5

I’ll talk to my bosses, but no guarantees. We’ve got agreements with various telcos, and I’ve got no idea what those contracts allow. Probably fine, but what do I know? I’m only a developer… :wink:

#6

Okay, Monday morning here, so I’ve talked to the Mangling Dictators of the company :wink:

They aren’t hugely comfortable with the idea of sending our source code anywhere, but happy for me to spend some more time working on this.

If I provided you with the open source components of our application (jetty web server, plus our cut-down, roll-your-own app server) can you give me the build scripts required to get it working in Jet? I’ll then take the internal code (EJB’s, etc) and extend your build scripts to compile accordingly. At the end of the exercise I should be able to provide you with before and after results – basically page views per second under the different environments (with a rough description of the test).

Let me know if you’re still interested and what exactly I need to give you.

J

#7

[quote]Let me know if you’re still interested and what exactly I need to give you.
[/quote]
Sure we are interested. By the time we return from JavaOne (Monday July 5th), please prepare a complete system that works on HotSpot, and instructions or how to run it, check that it runs correctly and measure its performance (if applicable). Email me at dleskov at excelsior dash usa dot com when you are ready.

Dmitry