OpenGL.org: Xj3d now twice as fast without Java3D

General Discussions
#21

I do understand your criticism on the lack of Xj3D documentation. It’s something that I’d like to make better, but to be honest, we’re still not even at version 1.0 yet. Things are changing on a daily basis and keeping up to date docs about changing the internals is rather hard to do.

We basically have two main classes users of Xj3D - those using either the SAI or EAI to create a browser and work with it at the X3D/VRML spec level, and those using it as a Java3D loader. It’s extremely rare that we see anyone using it any other way. As such, there has been no demand for external documentation for doing other work that is down in the weeds with it. In both cases, the usage documentation is well specified elsewhere.

If your interest is in creating new nodes or changing something about the implementation, then there’s documentation on the Xj3D website about how to do that. Properties that can be used to control the functionality are documented on the front page (overview) and in each package overview.

Apart from those, what else do you want to see in the way of documentation?

As for things not working, that’s correct, we have bugs. Sorry, but the X3D and VRML specs are horribly complex things to implement. X3D is now three separate specifications with over 2000 pages of spec docs. Another thing that we run into time and again is that it’s not our bugs, its the user’s VRML files that are not spec compliant. Blaxxun and Cortona are notorious for completely ignoring the specification and letting all sorts of stuff through that is illegal. OTOH, we are strictly compliant to the spec and will refuse to run a file that is not spec compliant and point out the error. Are you sure you haven’t confused that behaviour with a full-on crash?

Just to explain a little more - In earlier versions of Xj3D, whenever we got errors we would report these as exceptions from the guts of the nodes. However, our output code like the consoles weren’t particularly intelligent about how they handle them and would dump stack traces for stuff that really wasn’t a genuine error and some of the reporting paths for errors were not hooked up. Something like a colour value out of range would still end up with the complete exception trace being dropped to the command line. M9 and some recent work post M9 have pretty much removed all of those now and will just report the correct error and line of the file it occurred on.

As for the lots of jar file comments, well that’s a personal preference and one which I don’t agree with. Since Xj3D is designed to be segmented and used in parts, our users prefer having the ability to take just the JAR files they need rather than one huge file that is around 4MB in size. If you look at other projects of similar size, such as Batik or JBoss, you’ll see they take exactly the same approach as us - a lot of smaller jar files that allow the code to be segmented and only the appropriate parts needed. For reference, the Xj3D library is now over 450K lines of code. Add in all the external libraries like j3d.org, URI handling etc and we’re well over 700K lines easily. Aviatrix3D I did a quick wc -l on last night and it was 75K lines alone!

Edit: As for the javadoc being pitiful comment, I reject that completely. Xj3D has the best javadoc of any project I’ve ever seen. You’re telling me http://www.xj3d.org/javadoc/ that that is poor javadoc, then I’d like to see what you classify as “good” javadoc. Considering every single method variable and class is commented with more than just “sets the size” sorts of comments, then I fail to see what else could be done, unless you’d like us to comment it in Z to formally prove the function of each method call!

#22

It does, at least, have package comments. Which puts it in the “better than average” category. Modulo your comments above it sounds like you’re not aiming for much documentation anyway, so perhaps for your situation that’s very good.

However, my rule of thumb rating for jdocs of generic projects claiming to have good ones would rate you quite poorly. I take the average number of more-than-one-sentence paragraphs in the package comments and mulitply by 20 to get a percentage score. On that basis, you get somewhere between 20% and 40%. Not great. But the sad truth is that I’ve used software with 7-figure licensing costs whose entire documentation was jdocs which would only get about 10% to 30% on that basis. Wasted about half the development time just trying to invent our own documentation (although there wasn’t much wrong per se with the jdocs, aside from the odd class [our of several hundred] whose method docs were useless)

#23

[quote]But the sad truth is that I’ve used software with 7-figure licensing costs whose entire documentation was jdocs
[/quote]
Wow, what Java based software has a 7 figure license? :o

#24

[quote] However, my rule of thumb rating for jdocs of generic projects claiming to have good ones would rate you quite poorly. I take the average number of more-than-one-sentence paragraphs in the package comments and mulitply by 20 to get a percentage score. On that basis, you get somewhere between 20% and 40%. Not great. But the sad truth is that I’ve used software with 7-figure licensing costs whose entire documentation was jdocs which would only get about 10% to 30% on that basis.
[/quote]
What is the basis for your “rule of thumb”?

#25

[quote] However, my rule of thumb rating for jdocs of generic projects claiming to have good ones would rate you quite poorly. I take the average number of more-than-one-sentence paragraphs in the package comments and mulitply by 20 to get a percentage score. On that basis, you get somewhere between 20% and 40%.
[/quote]
That’s a pretty poor metric. That’s like reading the first 5 unnumbered pages of a book (ie the copyright notices, printer information and acknowledgements) and determining whether the writer had a good story. Your metric uses an entirely arbitrary mulitplier for something that is already an average value. If every package was documented with 2 sentence paragraphs, then you’d end up with a rating of 200%, which makes no sense at all. Most of the useful information is not at the package level but in the individual classes. If you applied that metric-style analysis to the class headers rather than the package docs, I would agree that it’s a pretty good determinant of how well documented a project is in javadoc. Packages, because they contain a collection of very loosely affiliated sets of functionality aren’t really a good way of documenting code. The usage of the individual class, however, is where the good value documentation can be found.

Though, it is a funny thing that you talk about that. Each time I write one up, I always try to put a lot of text in there, but usually I come up with a blank as to what exactly I should be saying ??? The short ones are not for lack of trying to put something useful in there, that’s for sure!

Mojomonkey - anything like the big J2EE app servers could easily run into that sort of licensing costs. Also, I could see the Oracle licensing being up there as well for a big site.

#26

DB stuff…dig around in the enterprise computing “niche” and high software costs are not uncommon. IIRC DAoC (an MMOG) was quoted $800,000 for their Oracle licenses during development…

#27

It’s not scientific, it’s based on experience. If I’d said “this is the formal metric company X uses” then it would be ludicrous, especially seeing as it can be easily gamed. (although obviously I use some common sense - clamp it to 100% and you would need an average of 5 x 2-sentence paras per package to get that)

The point is that I tried to find what was in common between the well-documented stuff I’d used, and what distinguished effectively inside the grey area of “OK but not great”, and even the best documented stuff has the odd package with just a sentence or so (c.f. below).

If I were attempting something more formal I’d do a weighted score of package and class comments - I agree entirely that the class comments contain substantially more information, but there are two lessons from experience:

  1. People who document poorly NEVER write package comments; people who document sort-of-OK almost never write package comments, people who document well typically write package comments. They are a quick-n-dirty (hence, rule of thumb) way of estimating the doc quality.

  2. It’s bloody hard to visually/manually rate class comments for non-trivial projects in trivially short time. It took me a minute or two to analyse your package comments - how many hours would it have taken to look at every class? - although, obviously, if you were doing something more formal you could (and would) put it all in an automated script (wahay!). Indeed, I’ve used java-centric testing tools in the past that do exactly this (but with more detailed heuristics for scoring) as one of their measurements of how “complete” a class / package / application is (lots of different metrics for: how good is the unit testing? how good is the documentation? etc)

Yes. BUT…most of the useful information for the user to generate a mental map of the system/software/library is contained in the package level.

If your software is only used by people who already know how it works, and just need your implementations docs (e.g. OpenGL) then there’s almost no need for package comments, beyond one/two sentence descriptions. Indeed, most multi-pkg software seems to end up with a couple of such packages IME - and this may explain the situations where you couldn’t think of much to say; e.g. I encounter similar stuff when doing networking systems: a pkg dealing with a protocol doesn’t need much package commentary. E.g. a codec for HTTP I’m not going to explain the concept of requests, responses, HTTP pipelining etc - I just put a package comment “uses the same terminology of RFC XXXX”.

Good documentation is hard and time-consuming to produce. The time isn’t taking writing it, it’s working out what to say. c.f. above sometimes there just isn’t much worth writing (without duplicating the text of a formal spec, which would phrase it so much better than you could!). But if you often have much trouble with a pkg comment then perhaps your pkg is not as well pkg’d as it could be, and needs refactoring, or perhaps you just need a bit more practice of writing user-level docs (so that it comes more naturally) - it seems that most programmers today get little or no practice at this stuff, so why should they expect to find it easy?

#28

Fair enough. My testing tends to be based on random sampling of the javadoc - click in each package and look at 2 or three of the classes and packages. Usually you get a decent idea just from looking at the overview to see how many classes have class docs and then at the methods and variable summaries.

One thing to note with the Xj3D javadoc is that we only run javadoc on about 30% of the packages. All the individual node implementation packages we don’t both running as we think it’s wasted effort and makes it harder for a user to make sense of what is going on. We try to cut down on “over documenting” because it leads to more confusion rather than just running javadoc on the essential parts that an enduser that wants to extend Xj3D would use. 60 packages of “This implements the Box node in OpenGL” is useless to almost everyone, including ourselves (I use my own Javadoc as I develop too as the codebase is so big I forget what a lot of my own methods do, so I document for myself as much as for anyone else).

#29

[quote]Edit: As for the javadoc being pitiful comment, I reject that completely. Xj3D has the best javadoc of any project I’ve ever seen. You’re telling me http://www.xj3d.org/javadoc/ that that is poor javadoc, then I’d like to see what you classify as “good” javadoc.
[/quote]
Well what i wanted to say is that javadoc is just the minimal documentation being done well or not. My sugestion is to document pre/post conditions and invariants on the interfaces or classes if you dont provide an interface for that class. The profs at my univ recomend we use iContract for generating code to validate preconditions. And they also recomend we use UML extensively not just the class model diagrams.

The reason why i mentioned the multitude of jars is because java1.5 already comes with much of the funcionality you get from the extra apis you include in xj3d. So my guess is all the extra apis you bundle with xj3d could be replaced by java apis and reduce the distribution size.

This is important because if a person wants to make an apllet to display x3d forcing the guy who is loading the applet to wait several long seconds (minutes) is very bad. See the size of blaxxon plugin. Maybe this is not your development target but a small size applet is essential for a x3d viewer.

I have downloaded M9 and tried it. I think there is a problem in your full install executable. I have used the installer but at some point he tells me to delete old xj3d and to press ignore on the dos window prompt that follows if there was no previous instalation. Man using dos prompts with a nice install wizard interface ?? Then it installs a javavm and asks me to reset but after reseting, the installation process is forgoten. I was able to install xj3d by answering no to reboot after installing the javavm again.

I only have a simple request for you Mith abolish every mixture of java and msdos in your applications. Msdos is not like bash. Using msdos is even worst than using qbasic mixed with java. :wink: Right now i cant even launch the browser on xp the msdos version of xp has been severely tunned down by Micro$oft.

There are so many other possibilities. You could use a .ini file as a command file; a bash version compiled for windows; an executable to launch your class with an .ini file to setup your classpath; a .js file (windows .js); a java class file; a jar with a properly setup manifest file; use java web starter.

ANYTHING BUT MSDOS. P ! L ! E ! A ! S ! E !

#30

[quote] Well what i wanted to say is that javadoc is just the minimal documentation being done well or not. My sugestion is to document pre/post conditions and invariants on the interfaces or classes if you dont provide an interface for that class.
[/quote]
That’s there already (mostly). Everything I write javadoc for has that information already written. I’ve been coaching (more like hassling…) Alan and the other coders into provide better detailed javadoc, but they’re much more hacker oriented than I am.

[quote]The profs at my univ recomend we use iContract for generating code to validate preconditions. And they also recomend we use UML extensively not just the class model diagrams.
[/quote]
This works well up to a point and is nice in theory. We already have large-scale architecture and runtime UML documentation in place. Look under the architecture docs on the Xj3D site and the docs/arch in CVS. The large-scale architecture of Xj3D hasn’t changed in 2 years. Some of the details have, but not the basic philosophy and runtime interactions. Keeping detailed architecture docs up to date with our day to day coding is close to impossible. Last time I ran a project this side, I had 15 developers working under me, it it was far easier to keep more detailed docs current. Right now, there’s myself and one other fulltime developer, and one part-timer. Since the large scale architecture is unchanged in such a long time, I see no need, nor do we have the resources to maintain detailed parts.

[quote] The reason why i mentioned the multitude of jars is because java1.5 already comes with much of the funcionality you get from the extra apis you include in xj3d. So my guess is all the extra apis you bundle with xj3d could be replaced by java apis and reduce the distribution size.
[/quote]
Nice in theory but doesn’t work in practice. We have some of our users that we’re attempting to drag off 1.3 right now. They’re fighting kicking and screaming all the way too. Moving to be solely 1.5 is an impossible pipedream for the next 3 years or more. The only reason we’re pushing these users to 1.4 is because JOGL has it as a hard requirement and they want to use that for the renderer for the better performance and not use Java3D.

Also, the stuff in JDK 1.5 does not actually support about 3/4 of what those JARs do. URI handling is a joke, so is the extended protocol handling (particularly URL handling that has to do more than just fetch a static webpage). The regexp in java.util.regexp is so buggy as to be non-functional. The only other library jars that we bring in from other projects are the j3d.org code and Aviatrix3d, both of which JDK 1.5 doesn’t do :slight_smile:

[quote] This is important because if a person wants to make an apllet to display x3d forcing the guy who is loading the applet to wait several long seconds (minutes) is very bad.
[/quote]
That is of no concern to us as we are not targeting that audience. It would be impossible to use Xj3D in any applet form without installing it first - either because of the JOGL/J3D requirements, or because of all the other non-safe stuff we do down the bottom, such as messing with threads, external network handling, property configuration etc.

[quote]I have used the installer but at some point he tells me to delete old xj3d and to press ignore on the dos window prompt that follows if there was no previous instalation. Man using dos prompts with a nice install wizard interface ??
[/quote]
It’s actually a native C app that is there to deal with some of our really old users that still have stuff installed in the wrong place. It’s quick and dirty hack, as we’re not Win32 developers, that is not intended to remain long term as part of the installer. The intention is to leave there for about another 12 months or so until we can be reasonably assured that there are no more pre-M7 users floating around out there. If someone wants to spend some time in VisC++ to write us a sexy version of the same code, that would be cool, but it’s not really of any interest in us doing so. However, it has to work for everything from Win95 and upwards, hence the simple pragmatic nature of the current setup.

[quote]I only have a simple request for you Mith abolish every mixture of java and msdos in your applications. Msdos is not like bash.
[/quote]
Well, in the way that we have to use it, it is perfectly suited. It’s just a simple launcher window that happens to integrate better with the OS than any of the other suggestions you’ve made, with the exception of the native launcher. All the others have problems dealing with drag and drop starting, controlling Java commandline parameters like heap size etc.

We could go for the full-on native launcher, but right now that hides way too much important information for us. We’re still not rock-solid stable, and the commandline allows our users to quickly drop back stack traces with minimal effort. In order to make a better toolkit, it’s more important to us to do it that way than to hide the code right now. In addition, most of our users, don’t use the standalone (dos-prompt) code. They use Xj3D as something integrated into other applications, typically through the SAI/EAI so it’s less of a pressing issue for us. The standalone browser is almost an inconsequential afterthought to us as not many of our users want to use a standalone browser.

Then, there’s the other part of this - we do stuff we’re paid to work on. Considering we have a large amount of paid work, anything like that, which is inconsequential to the daily usage of the code gets pushed right down to the bottom of the priority ladder. If someone wanted to pay us a few grand to spend a week developing and testing a nice native launcher app that integrated with a given operating sytem, then we’d quickly include it (or if they wrote one and asked for it to be included in the CVS code and agree to maintain it). The reality is that nobody has, nor do any of our existing clients show any interest in such activity, so that’s not likely to change right now.

[quote]Right now i cant even launch the browser on xp the msdos version of xp has been severely tunned down by Micro$oft.
[/quote]
Considering that our development machines are all XP and a couple of Win2K laptops, which it works on just fine, I would say you’ve got something messed up in your base environment setup. Perhaps you’ve accidently reduced the maximum command line length in numbers of characters or something like that.

#31

Do you have details on this? I’ve mostly migrated from OROinc to util.regex now, and although I’ve not encountered any problems yet it would be nice to be aware of any mines…

#32

User that persist on using 1.3 ? Considering that the sun jdk is free to download those guys have to be very stuborn people.

Thats too bad. Still with jogl it may be possible to use xj3d to prototype game ideas.

Nope I avoid messing with anything msdos related.

#33

Im not shure about this but if you try disassemble 1.5 proprietary class files into java source code you may notice that the gnu java regexp is in use there together with a lot of apache code for working with xml. The java.util.regexp api may a wraper to the gnu regexp.

I used jEdit with jInsight plugin to check this, but then again i am not shure i have used the program properly.

#34

[quote] User that persist on using 1.3 ? Considering that the sun jdk is free to download those guys have to be very stuborn people
[/quote]
Unfortunately, that’s life for commercial software development. They have systems that have a known good baseline and want to maintain it. Also, consider that it was only 6 months or so ago that SGI released their JDK 1.4 port and you should be able to get the feeling why this is the case. Although we would love to run with the newest features all the time, clients like to have stable baselines to work from and not have to upgrade their software whenever we drop a new dev release (which is about every 2 weeks currently). Like yourself, we are still getting bug reports from one user about M7 code, which is over 12 months old now.

[quote] Thats too bad. Still with jogl it may be possible to use xj3d to prototype game ideas.
[/quote]
Everyone else seems to be more enthusiastic about this concept than I am. My personal opinion is that VRML/X3D is quite a poor choice for prototyping games beyond puzzle-solvers. I must be wrong though as a lot of people like the idea. We are JWS capable now thanks to some work by one of our contributors, so at least that part is a start. I also spent 3 weeks trying to build Xj3D as a native plugin to Mozilla courtesy of the Blackwood code, but that failed rather miserably :frowning:

As for the JOGL side, I will be writing a loader to suit the aviatrix scene graph shortly. I’d like to do something that works for those that want to play right with the raw GL commands, but don’t really know the best way to go about it. That’s a low-priority requests from one of our clients, so if anyone has some ideas on what they’d like to see in that regard, shoot me a PM.

So, in summary - could we do better in documentation? Most certainly, I think it’s never possible to not be able to do better. We’re about to go through another round of internal API changes again across the whole codebase, so time to go back and check out everything we have. If you guys have any areas that you would like to see better documented, drop me a line with what you’d like looked at and we’ll spend some time there on it sometime just after Siggraph.

#35

I’d be interested too, because as old GAWK user I use java.util.regexp with pleasure and did find it to work fine for me. But then, I’m unfortunately just using Java in my spare time :frowning: so full time Java developers usually know more details.

#36

No problems with regexp here, although I’m currently only using it in one slightly-bigger-than-small personal project.

#37

It’s been about 12 months or so since I last played with the portion of the code that does regexp (it’s part of one of my other utility libraries, not Xj3D/Aviatrix3D/j3d.org). It was about the 1.4.1 timeframe and I’ve not revisited it for 1.4.2. Thus my memory of the exact problems is not that precise (meaning I’ll probably have someone show me the errors of my ways :slight_smile: )

I mainly use it for pulling apart an arbitrary ruleset for parsing resource location information for a generalised URI library. I have configuration files that specify a resource subsystem component with a given name. Then a regular expression is defined for how to take a user-provided string and rip it apart to locate various pieces of vital information. For each resource subsystem, the string format is extremely different in it’s syntax. This requires setting up some semi-fixed formats for the actual regular expression (delimiter types, sub-expression numbers etc).

The JDK regexp library had a lot of issues dealing with the arbitrary input of the regular expression - particularly dealing with finding non-alphanumeric delimiters in the provided string. For example, it would have issues is the delimiter was an ampersand character rather than a backslash. Another area I vaguely recall it had problems dealing with was the escaping of characters being put into the strings to do the subexpression parsing with. Sometimes it would just arbtririly decide the regular expression was invalid, other times valid - for an identical input string. The JDK version seemed to have no consistent behaviour in this regard. So, while it seems OK if I have a known, fixed expression that I want to match with, for arbitrary setups read out of external config files where there was no control at all, it would fall over quite miserably, while the GNU libraries had no issues at all dealing with identical input.