Since (micro-) benchmarking is a science on its own, I thought let’s ask here.
Say a person not knowing anything about the two languages asks you to do a comparable benchmark in Java and in C++, because he doesn’t trust any external quotes (just what he sees). He’d like to start a C++ exe and a Java Jar/Bat file on the same PCs (several actually) to see the differences.
It should be a small and easy to implementable game related benchmark, including computing operations (needed for simulation) and such like.
Are there any benchmarks sources doing something comparable? If not: how to write some short one which doesn’t break any of the many “don’t do this ever in micro-benchmarking” rules…?
(Some days ago a new benchmark doing similar things have been quoted here in the forum, comparing 9 languages. It’s nice basically. Aside the trigonometry numbers for Java 1.4.3 - when does this problem being solved?)
I know such questions tend to sound very silly. However, just imagine for a second you would have been asked to do this exactly: what would you answer/do ? Thanks.
[quote]what would you answer/do ?
[/quote]
Slap them, with a large fish.
If they persist:
Get them to detail what they expect the average frame to have to do.
i.e:
a) 15% complex float maths (physics)
b) 10% traversing bizrre linked data structures
c) 20% Game logic (mainly ifs/some integer flag stuff)
d) 5% sound & effects
e) 50% drawing sprites in software
(or e) - rendering sprites using whatever API you want)
then make some simple test cases that test the critical aspects of each task:
a) do some complex maths: a matrix by matrix multiply, a convert to quaternion, a vector normalise, and a 5 elemnt taylor series expansion.
b) build a linked list of derived classes and call a virtual function on each. Then delete the whole thing.
c) Generate a random number, then run through a series of if statements testing each bit, some statements using &&, some ||, etc. And a 8 case switch statement.
d) Play 8 sound effects, each one starting and stopping. And stream a large sample as well.
e) This one is mainly an uncached memory read/write to array problem, so make a large array (640x480, but 1D), and write to random sections (32 x 32byte lines to simulate a sprite - each line of the sprite would cause a cache miss). The trick is to get around 2x screen overdraw for realism.
Then build a test case that runs each test (while sound is going on, as sound is more of a %CPU usage as opposed to time taken).
Time for AT LEAST 1 MINUTE.
Write up the results, then slap the person some more for wasting your time when they end up doing something completely different to what you tested anyway (which they will do).
Dom
PS: If they are serious about writing the games to make money, you must consider that you will have to use Java 1.1 or you will lose 75% of your visitors. If using C++, use OpenGL for graphics, as this is much nicer to port.
PPS: Unless you really want the run-in-a-web browser (which I do for our demo versions) use C++. Being able to use pointers, Intel compilers, BoundsChecker & Debuggers will save you much pain for a large game project. Dealing with hardware wierdness is nothing compared to dealing with Java VM version wierdness. Plus, a real publisher (CD’s in boxes type) will not take a Java version seriously. Sad but true.
(snip excellent advice: get a requirements list off them, and then implement it).
I suggest you also implement if all within Unit (C/C++) and JUnit (Java). JUnit is a bit pants these days (ONLY because of the lack of tools that a “real” testing suite would come with as standard), but… there are some good “add-ons” for JUnit which do a lot of this.
e.g. I recently spotted an add-on which effortlessly times all your tests, presenting results individually as well as in aggregate.
e.g. I have seen (although I can’t find it any more ) a nice simple visualization tool that automatically finds and runs all your JUnit tests, and presents a summary in the format of a typical “report”. But you could pretty easily write a small app to auto-translate JUnit output to an HTML file
Couple of advantages to doing it with (J)Unit:
[]It doesn’t mandate much, but at least provides some nice consistency and formalism to your tests
[]the Unit stuff is so widely used in commercial programming that any new staff have a very high chance of already having used it (and I’d hope that the people you’re trying to convince have used it at some time before in their lives; should make it slightly easier for them to read, trust, and fiddle with your source to satisfy themselves)
[]Gives your team an example of Unit tests before you even start the project (it’s easier to get people to write them if they’ve already written / used some)
[]It’s a good start - you could encourage people to add extra tests (because Unit makes adding tests very simple; it picks up new tests automatically without having to be reconfigured/recompiled each time) for any new things they are suspicious about
I’d say at least one hour.
Export your results (or save them directly) as CSV. Then everyone can open them in Excel. This is handy, because Excel can instantly and effortlessly produce all manner of graphs, statistical analysis, and it’s damn easy to sort stuff :). I hate Excel, but basic stats analysis takes < 30 seconds in excel (plus loading time ), so I confess to using it quite often :).
Ahh, I’m one of the disagree-ers here…I say “mandate 1.4 and be damned”.
Bear in mind, crystalsquid, he’s talking about a 2-year dev time. In that period, the installed base of 1.4+ is going to sky-rocket, largely because of the pre-installs on all Dell and HP machines (IIRC; yes?). Simultaneously, there will be more upgrades/replacements of old versions simply because 1.1 is now removed (IIRC ?) from windows, forcing more people to download a sun version.
Also, and this may be a very small thing, java is now so old people are starting to trust it more :). My own brother, who refused to install java since his last re-format of windows (oh, the joy!) because an old JVM (1.2.x JDK) added crap to his computer and didn’t install properly, has finally acceded; like Flash and Shockwave, the longer Java is around, the longer the world goes without “Java virus destroys world” stories, the less suspicious joe bloggs home user becomes.
But I liked your list of things to test, it’s a very handy starting point.
[quote]Since (micro-) benchmarking is a science on its own, I thought let’s ask here.
…
Are there any benchmarks sources doing something comparable? If not: how to write some short one which doesn’t break any of the many “don’t do this ever in micro-benchmarking” rules…?
[/quote]
Also, if you do go the Unit route, if you are willing to put the code somewhere public (perhaps on JGO?) I would contribute some mini benchmarks too; there’ve been a couple of times in the last 12 months when I wanted something similar (although usually to test 3rd party libraries in conjunction with games dev, rather than test java vs. something else) but knew I’d never have the time / effort to write anything comprehensive enough to be worth it.
If you got something started, so I could just come along and knock up e.g. some simple NIO tests…
Many thanks for the good input, guys. I’ll have to re-read your articles a few times and think about them.
Well, I think I’ll start with the … fish.
JUnit - never used but will start reading an idiot’s guide to JUnit.
Yes, indeed, I forgot to mention the project would be using 3d, so some 3d API will be there and no software sprites this time, luckily.
Blabla is right: it’s for a larger game, too, so no Applet, plus Java 1.5 should well be ready then. Java 1.7 maybe? Size doesn’t matter: it will be on CD (in case the publisher will let us - I think I’ll take a copy of Chrome game with us, plus a big fish).
[quote]PPS: Unless you really want the run-in-a-web browser (which I do for our demo versions) use C++. Being able to use pointers, Intel compilers, BoundsChecker & Debuggers will save you much pain for a large game project. Dealing with hardware wierdness is nothing compared to dealing with Java VM version wierdness
[/quote]
C++ sucks , 2) pointer sucks , 3) Intel’s compiler is cool but too many Win32 games can’t use it because they’re contaminated with M$ libs/etc which won’t compile on Intel’s compiler, sigh (ISO C++ standard? Not for Win32), 4) hardware and software (Windoze) weirdness is such a pain, not again thanks, and since it would be using one JVM version: nice. 4) Distributed compiling is a nice idea but not handy at all.
Etc blabla. Anyway thanks, Crystalsquid, for the nice benchmark framework ideas!
So, a lot of work. Maybe for nothing. Well, no risk no fun. Still Java is really nice.
[quote]Unless you really want the run-in-a-web browser (which I do for our demo versions) use C++. Being able to use pointers, Intel compilers, BoundsChecker & Debuggers will save you much pain for a large game project. Dealing with hardware wierdness is nothing compared to dealing with Java VM version wierdness.
[/quote]
I didn’t come across Java VM version weirdness that much. And God knows how much I cursed those C/C++ typical hard to track down bugs.
Preston’s argument is true; if you ship on CD, you will ship with 1 JVM. You aren’t trying to compile your C++ code with 20 compilers, are you? (shaky anology, I know ;))
For me, java is a way to move forward in games development, even if it’s not all perfect for it yet. Like java’s progressing the last couple of years, it’s getting there.
[quote]PS: If they are serious about writing the games to make money, you must consider that you will have to use Java 1.1 or you will lose 75% of your visitors.
[/quote]
I’m a disagree-er too. Ban 1.1! 1.4 at least.
If you’d say to them “hey, look at my cool java game” you’re probably right. I’d say, show them your stuff and tell em it was java when you sold your first million ;D
[quote]I didn’t come across Java VM version weirdness that much
[/quote]
Oh - I have Using BufferStrategy causing it not to work on Macs, playing samples using 1.1 interface runs REAL slow on 1.4, etc. etc. mind you, we also had a ‘destructor in base class not virtual’ causing a bizarre memory leak just yesterday in our C++, so not much between them really I suppose.
[quote]I’m a disagree-er too. Ban 1.1! 1.4 at least.
[/quote]
It would be nice, but from putting our games on the web we found the 75% of people did not have 1.4, and would not upgrade (not surprising given the 15MB or whatever it is at the moment). Having said that - if there are lots of CD based games that people install which ends up with them installing a newer Java version, then this can only be a good thing. I know many people only upgrade their DirectX when they buy a game that installs it for them.
[quote]If you’d say to them “hey, look at my cool java game” you’re probably right. I’d say, show them your stuff and tell em it was java when you sold your first million
[/quote]
yeah, I disagreed with myself in the thread in the business forum. I used to advise my old company about some purchases (it was a very big publisher/developer) and to be honest, we didn’t look to close at what technology they were using if they had a decent demo of the game running. If it meets the golden rule - put CD in, install, press 1 button to play, then they don’t really care. If the install process is too complicated you will have issues though.
I am a die-hard C++ fan (well, assembler really), and my main issues with Java do stem from having to use 1.1 at the moment to ensure enough people can play our demos. (Grr!)
[quote]Well, microbenchmarking really isn’t a science, and the best answer is not to microbenchmark but macrobenchmark and profile a real problem.
[/quote]
Well, yes, that’s right.
If I understand Jeff correctly, then maybe I should have said: “Understanding what you’re microbenchmarking is a science” or such.
[quote]Oh - I have Using BufferStrategy causing it not to work on Macs, playing samples using 1.1 interface runs REAL slow on 1.4, etc. etc. mind you, we also had a ‘destructor in base class not virtual’ causing a bizarre memory leak just yesterday in our C++, so not much between them really I suppose.
[/quote]
Of course there are problems between java versions. To be fair, 1.4 on mac is fairly young; fullscreen on windows didn’t work flawlessly in the 1st versions. The 1.1 sound problem was a bug which is probably solved in the meantime (haven’t checked though). But sure, there are problems and sometimes really frustrating ones.
You have strange version issues between C compilers as well. I’ve seen some really weird workarounds to optimizer bugs for example.
The thing is, with java when you arrived at the point you come across problems on macs and linux versions, you might not even be halfway the windows version alone using C++.
Might be because I’m a pretty shitty C programmer these days ;D but I can program a hell of a lot faster in java with fewer bugs.
[quote]It would be nice, but from putting our games on the web we found the 75% of people did not have 1.4, and would not upgrade (not surprising given the 15MB or whatever it is at the moment).
[/quote]
I agree in such situations you have to just do the most practical thing. But besides that, I think it’s really worth it on the long run to push new java versions anyway. I’ve said it before and I’ll say it again: Make your games the large JRE download worthwile. (Not that I’m saying your stuff isn’t! Just talking in general)
Sticking to 1.1 will surely kill java for games eventually. It’s all up to us developers.
No - don’t add byte code stuff to Java… use one of the many other things that will produce Java bytecode. Don’t pollute the Java language… use an appropriate language.
There are things like BCEL (Byte Code Engineering Library ?) and other tools that you can use for these niche applications.
I know I didn’t see a DirectX 3.0 game for ages. So what the idea of outdated beta of Java?
If you make users lazy and expecting you’d do job for them, you’d be sorry for it quckly.
I think that best idea is to say: “You should download last version of JVM form here, or here. Current best version is xxxx” And add binaries compiled to one version backwards to alow user one year of time for updating.
It’s just about forcing users to do “uncommon things”, like regular update of theirs software. (Like antivirus, firewall, and so on) Actually number of e-mail viruses each mouth shows that “JOE” average isn’t aware about
his necessity of such things.
Java 15-44MB/year. AV 3MB/month. FW 5MB/ 3months.
Average DL of Dial up user at least 350-600 MB/month.
Conclusion If someone don’t have updated JRE it’s
Problem with lazy user.
User that didn’t know what he should do.
User that is scared with doing new things.
solutions:
Force him as mercilessly as creator of DirectX. You might have some problems, but it’s better in the long run.
Web pages for dummies is often best solution, just try to write it simple and provide user with step by step instructions. Don’t forget to add a warning. “you should read all text on this page, don’t forget to scroll down”. I know it’s silly, but…
Words “computers are changing and you need to upgrade to get new (faster) features”, or "some programs are demanding to upgrade your OS, we just need for you to upgrade your (Sun, or IBM) Java VM. "
(I actually thing that 3. problem comes with age. Users are used to old DOS, NIX or whatever and whe they should be doing something they they didn’t did before they try to move it to sometime later. Aka scream from linux boards: “DO ME A FAVOR AND UPDATE CLIB(don’t recall proper name). It will solve all of your problem without any stupid hacks. You know it’s freely avilable here right?”)
It’s just about thinking of reasonable, or completely unreasonable, but nicely sounding excuse. (More stable programs, and less necessary restarts excuse works well.)
As for microbenchmarking. If you know some methods from your game you could use them with real word data. However, it will be a big problem to rewrite them in C++ and ASM with high enough precission.
Why is Jeff so ???, when someone would start to talk about benchmarks?
Gee I agree with CS on everything except the PS which I have strong disagreements with…
I question the scientific accuracy of the 75% figure. More generally this is only an issue at all if you are targeting the web Applet market as it exists right now. If you are targeting either Java Web Start or Java Applications it a non-issue. I
f you are targeting Applets a year or two from now the situation is likely to be radically different as MSFT is due to STOP shipping 1.1 soon and we already have comittments from the PC vendors to ship real VMs with their systems that shoudl cover 50%+ of all new PCs.
(About the only reason to target Applets IMO is to reach the causal market today, for all other markets the other approaches are much more flexible.)
As Bioware discovered, just usiong OpenGL however doesnt gaurnatee yo ua fast or easy port of C/C++ code.
Absolutely 100% disagree. A competant Java coder puts out lkess bug-ridden code in anywhere between 2 and times the speed of C/C++ coding. I really need to scan that sheet of statistics from CGDC two years ago and post it. All the Java developers reported:
90%+ of their time was spent in development, less then 10% in Q&A.
Porting times to Mac, Linux and Solaris that ranged from 3 hours to 2 weeks for pure Java, and less then a month (PC to Mac) for Java ontop of custom graphics and Io bindings.
Over-all development times unheard of in the game industry. (ex: Shawn’s entire FullSail 3D game engine took less then 2 man years. Based on that engine he brought an initial FPS to demo playability in a month. He did a complete playable fighting game in less then 3 months, as I recall.)
Again untrue. Thre have been about a dozen preofessionally, in a box, published games partly or fully in Java in recent years. Roboforge (99.9% Java, 3 lines of C code) made it to a box. Star Wars Galaxies used Java for its scripting, as did VTM (both published y major publishers in a box.) Law and Order (100% Java) was not only published bya amjor publisher, but so successful that the publisher published a second game on the same Java engine.
Will you have to answer a few questions about “why Java?” probably. But it HAS been done successfully already.
[quote]Why is Jeff so ???, when someone would start to talk about benchmarks?
[/quote]
NOT benchamrks. Benchmarks are vital to performance coding.
MICRObenchmarks. Microbenchmarks are often misleading and more often then not useless. The problem is that microbenchmarks don’t act like real code so the question they end up answering 99% of the time is NOT “how will real code in this environment perform” but only “how does THIS BENCHMARK in this environment perform.”
Modern execution environments such as Hotspot are very complex and are designed to analyze what your progam is doing and attack it in the best manner for that problem. Sicne the MB is a different problem, you get different results.
To write a good benchmark you need to really need to understand a number of things:
(1) What is your target app really going to be doing, in total?
(2) How will this environment optimize that work.
(3) How does this benchmark behave>
(4) In what ways are the optimizations that will be done for this benchmark similar or dissimilar to what will happen with native code.
A good simple example is that microbenchmarks are commonly written to dive into a single loop and sit ther for the whole run. Ths is VERY different from an app which will be calling other functions and different loops all over the place.
As it happens, those two conditions are treated differently by Hotspot because one (the MB) calls for on-stack code replacement while apps geneally don’t. Just recently here on JGO we stumbled across a bug in Hotspot’s on-stack replacement that led the bunch of us here on a wild goose chase for about a week-- all based on a micro-benchmark.
I was the first full-time hire on Sun’s JDK performance tunign team and worked heavily on the tuning of JDK1.3. I’ve got a ton of war-stories in this area. Remember how slow Java2D was in 1.2? That was the DIRECT result of micro-benchmarking leading the 2D team to tunign the WRONg things. Come to one of my periodic talks at conferences on performance tunign Java and you will get to hear the whole story.
The number of microbenchmarks I’ve seen at this point probably number in the hundreds. The number I’ve seen that actually produced information useful to an app developer can be counted on two hands.
(Baring getting to a talk, or even if you do, I’d also recommend Steve and my book for an easy relatively short read that will likely open your eyes in a lot of ways: http:java.sun.com/docs/books/performance )
To sum it up, the prooblem with most (micro)benchmarks is that the authors have no idea what it is they are really measuring.
To paraphrase the NRA
Benchmarks don’t fool programmers.
Programmers fool themselves by mis-using benchmarks
A microbenchmark is about 90% likely to provide meaningless results.
Whether those results are unreasonably good (such as Java optimizing out the entire benchmark, which has happened mor then once) or unreasonably bad (such as the cost-of-interfaces microbenchmark recently that ended up tripping over an optimzier bug that did not remove all redundant null checks in casting in the on-stack replacement case) is anyone’s guess.
If you want small benchmarks, then find chunks of reasonably seperable pieces of functionality of your app that you are sure will be significant and can implement and test indpendantly. Be careful of getting too small or it can again become fairly meaningless (ex: poly fill rate which at this point totally depends on the graphics card and not at all on the processor.)
Some good examples of things to benchmark might be pathfinding algorithyms, fft, etc.
FWIW, I think some good ‘macro benchmarks’ would be helpful.
I mean, one micro benchmark is posted on the internet after the other. Here everybody screams either “evil microbenchmark” or “see? java is really fast!” (depending on the result ;))
Actually that microbenchmark is a rather good one because it rather highlights a glaring performance deficiency in one small area. Individually, little deficiencies like this don’t make a big difference to the big picture but if you just happen to write the application that strings a few of them together (like, say, you’re writing a 3D game…) it all adds up to a crappy framerate. So perhaps someone should post a “bug” on the trig score?
[quote]So perhaps someone should post a “bug” on the trig score?
[/quote]
The current trig score is the result of fixing a bug (I posted the link earlier). To get faster trig you have to relax the specification or create a new FastMath class with a relaxed spec (anything goes?). I suspect that this would be resisted because many would use it even when they don’t actually have a performance problem at all. Alternatively ask Intel to improve the accuracy of their trig methods to meet the Java requirements.
That’s a head in the sand if ever I saw one. We don’t much care about VMs that don’t actually accurately reflect the common hardware out there today. If Intel is the lowest common denominator then it should be specified to match Intel’s standards. No-one cares about a standard which doesn’t work well on 95% of the world’s hardware.
Thanks.
) a nice simple visualization tool that automatically finds and runs all your JUnit tests, and presents a summary in the format of a typical “report”. But you could pretty easily write a small app to auto-translate JUnit output to an HTML file
Using BufferStrategy causing it not to work on Macs, playing samples using 1.1 interface runs REAL slow on 1.4, etc. etc. mind you, we also had a ‘destructor in base class not virtual’ causing a bizarre memory leak just yesterday in our C++, so not much between them really I suppose.