I’m surprised this hasn’t been posted yet (did I miss it?)
http://www.sys-con.com/story/?storyid=45250
It’s all ready been on /. for two days…
Coz it’s just another load of pointless bollocks again 
Cas 
There should be a study done on how easy it is to make a benchmark show whatever the hell you want it to by anyone who has an axe to grind.
EDIT: just found this: http://cpp.student.utwente.nl/benchmark/
Proving that you have to be a real dumbarse to publish anything that says X might be faster than Y, because it’ll take no time at all for someone to prove you wrong.
;D
On a related note, it does appear to have set off a bunch of blog-based diatribes from all and sundry. From those (quite-rightly) picking holes in his methodology, to ex-syscon editors having a go at the magazine (and each other) for linking/quoting from it. All most amusing. Do these people have nothing better to do? Have they not heard of “set phasers to ignore”…?
yeah… it wasn’t done very well… but at least it will get a few people to update their perception of Java, just by keeping the mud slinging and exposing them to new information.
[quote]EDIT: just found this: http://cpp.student.utwente.nl/benchmark/
[/quote]
Ah well that is actually quite favorable as well! So even after fixing a lot of the issues with the initial test it still shows Java as quite competitive. Still beating even the Intel compiler on 6 of the tests (!) and coming very close to it’s performance (still beating GCC with optimizations) on others.
Overall this is still good information for people enlightened enough to update their perception of Java. Always the best tool for the job, and this shows that Java is applicable to more jobs than many would think.
yawn
So what does the benchmark show?
It demonstrates that the java kitchensink is pretty well done, that you have to do all that stuff for yourself if you need it in c/c++ and of course that c++ is damn ugly 
Nothing new ;D
The pure fact that they (the C+±guys) are doing those benchmarks shows how scared they are about Java performance.
In former times, only Java-guys performed the benchmarks to show ‘we can compete!’. Nowadays, THEY (the C++ guys) do the benchmarks to show ‘look, we can still hold it!’.
This makes me feel very comfortable.
BTW, for FlyingGuns I never performed any benchmarks (how fast can I iterate a list?? pointless…). I just let it go and performance never has been a problem…
The only benchmark I have or need is 60 frames per second, nothing more, nothing less…
Cas
I just noticed that in the updated C++ code, created objects aren’t deleted which is kinda debatable isn’t it?
IMHO this really shows that it’s almost impossible to do an apples-apples comparison of 2 different languages using microbenchmarks.
[quote]I just noticed that in the updated C++ code, created objects aren’t deleted which is kinda debatable isn’t it?
[/quote]
Absolutely - they were mistaken about the concept of garbage collection. At the very least they should have put in a C++ garbage collector. They seem to think that since you don’t explicitly delete in Java then to fairly compare you shouldn’t delete the objects in C++ – idiots. (The garbage collector is doing the ‘deletes’ for us you fools!! That’s not the same as omitting them entirely!)
They are partly right in that it would be unfair to base comparison on a Java app that was so short that it didn’t need to collect anything at all. Java code should be designed so that the total space allocated over the course of the test is several times the size of the heap.
The problem with MBs like all this is at the end of the day they test … a program that does nothing. Imagine if you had an infinitely clever JVM and an infinitely clever C++ compiler:
java -jar test.jar
Execution time : 0.0s
test.exe
Execution time : 0.0s
because they don’t actually do any bloody work. What a waste of everybody’s time this benchmarking thing is.
Cas
[quote]They are partly right in that it would be unfair to base comparison on a Java app that was so short that it didn’t need to collect anything at all. Java code should be designed so that the total space allocated over the course of the test is several times the size of the heap.
[/quote]
Sigh…I agree. And have you actually tried that in any decent sized application ? I sometimes encounter GC pauses of 10s and above (on an Xmx of 300MB) and with just a few passes.
What we need at the very least is an order of magnitude more performant GC. Are the GC implementations (Sun, IBM, HP,…) the very best that are known to humankind or are there some exotic alternatives buried in the academia ? We need a selfless consortium of all the bigwigs in the industry to pool their resources to take GC to the next quantum level.
I’ve never had a GC problem since 1.3.x, and I’ve worked on datasets of several times the available memory; given the current variety of GC options available on the Sun JVM’s, I automatically distrust anyone who is having problems with them.
I’m not claiming you’re wrong in your case, but it would seem to me that the majority of people do not need a more performant GC, given how very few of the claimed GC problems in places like this turn out to be actually the GC at fault (most are something different; many are just stupid programming 
).
Could still do with escape analysis and Structs! That’d give us an order of magnitude more performance, and no mistake.
Cas
I envy you :). You said “variety of GC options”. I say that’s what makes me faint-hearted to try out all of them for a particular JVM version, conjure up a recipe based on my heuristics, and then distribute it as sacrosanct to clients with various configurations. I agree with you that it is a worthwhile exercise to adopt better coding practices to keep the GC quiet. But playing with the options…remember Sun has marked them with an “XX”.
Alright, things appear to be a little different with 1.5. In fact, Sun probably realized that the options are one too many for the mere mortals merriment, that they have this new adaptive policy. Haven’t tried that yet.
In my apps. (primarily sciviz, and a little into games - blame you all !), I don’t do any exotic creation of millions of objects. My requirements are mainly pretty humble - dynamically resizing primitive array lists (hand rolled by self since collections are just too awe-inspiring ). It is simply difficult to anticipate the size of the arrays - unless I go overboard. And I think that when I repeatedly allocate very long arrays (but not typically many of them) the GC appears to have some difficulty.
BTW, why is this Structs proposal still in the cold ? Sun should either evaluate it and ditch it or come up with a better equivalent.
[quote]Could still do with escape analysis and Structs! That’d give us an order of magnitude more performance, and no mistake.
Cas
[/quote]
Yeah, yeah, all my votes for structs are there already :
But what exactly do you mean by escape analysis? I only heard it has something to do with allocating on stack instead of the heap which supposedly would lessen the stress on the GC, but can you shed some more light on this? How would that help performance so much?
I think there were some words about it in http://www.java-gaming.org/cgi-bin/JGNetForums/YaBB.cgi?board=Tuning;action=display;num=1076185882;start=23
/M
My regular job involves datasets up to 1GB or so. If the task will need say 300MB of live data, then it helps a lot to set the minimum heap size to at least that. This avoids many cycles of increasing the heap by a small amount. The current GC do a full collection before increasing the heap size, and if you start from the default minimum size heap this will happen a LOT before you get your 300MB in the heap.
[quote]Could still do with escape analysis and Structs! That’d give us an order of magnitude more performance, and no mistake.
Cas
[/quote]
I would be surprised if you got anywhere near that level of performance increased except perhaps in unusual corner cases. Research papers on escape analysis rarely quote much more than around 20% increase. In the early days of Java when GC was very much worse, the gain would have been rather greater, but now with improved gc for short lived objects the value of escape analysis has declined.