Generic OpenGL UI Project?

General Discussions
#1

Lately, I’ve read about several UI projects in different forums, such as:

Fengui: http://fenggui.dev.java.net/
Squareheads gui: http://home.halden.net/tombr/gui/gui.html
Banana User Interface toolkit: http://samskivert.com/code/jme-bui/
JMEDesktop: (cant find URL to it right now)
Small Gui System (SGS for short) for Xith: http://www.java-gaming.org/forums/index.php?topic=12185.0
GUI3D: http://gui3d.org/

Now… all of the above projects are great on their own merits, but in general I believe that either their philosophy is wrong, their scope is too limited and/or they have other problems like performance and lock-in to a specific engine.

What would make a good UI system? What functionality must it possible possess to conform to my weird standards?

Ok, for starters, I have a few points which I would consider to be essential for a future-proof UI system.

  • Build for both opengl bindings (JOGL/LWJGL), not an engine such as Xith or JMonkeyEngine
  • GUI Markup in XML. The future of GUI’s is not in-code, so don’t mimic swing
  • Odd interface design should be possible, a-la winamp & co
  • Animation-markup. (both animation -of- gui elements as well as support for animated textures) Mouseover / appearance / dissapearance behaviour in XML.
  • Easy to get started with, but should allow for complex gui structures. I want Azureus to be on a wall in a maze, where you can shoot the buttons with your plasma gun.

So I’ve been thinking… of course, to start my own project, but that would be silly, with all the effort going on. However, to get every gui author to collaborate on something new, bringing in their knowledge… I can see the grimaces on their faces right now. ‘what? and abandon what we’ve done so far??’. Yet, this is precisely what I’m asking. But not to abandon your projects, but bear with me…

There already is something which would allow me to mostly do what I described above here… however, the scope of the new project will become immense… but the rewards evenly great… what I’m talking about is XUL. The XML UI Markup Language used by mozilla. (see http://www.xulplanet.com/ and http://www.mozilla.org/projects/xul/). The advantage to using XUL as a basis is that it is a proven concept. After the design phase it would be easy to assign diffent tasks to different developers. progress can be easily tracked, and we have tons of interfaces to actually use/test.

What do you think? is it a good idea? are there other properties you would like opengl user interfaces to have?

#2

I would love something like this.

The wurm gui classes mimic swing a bit, with class names like WList and WButton. It’s far too complicated to code for, and it doesn’t support the most important feature of all yet; displaying ui windows sent by the server.
Right now we use a JWindow with a html panel in it to show stuff sent by the server… and that’s really bad.

Does XUL support fancier gui components, such as treelists?
If this gets started, I would be willing to contribute code.

#3

XUL supports all that and more. But i don’t think it is both neccessary nor feasable to mimic everything what xul can define (XUL is only a standard). It would be up to the project to actually parse all that and implement renderers. By using XUL as basis, we can start with something simple, like labels, text, buttons. But since the scope of the project, the extent of the features, even the API and naming-convention is already done, it will be more straightforward than to come up with something new. But trees and tables are some of things I would consider low priority. The first goal would be to define the interface to LWJGL/JOGL, build the parser and get something simple running like a few buttons so people can start using it. If I hear many positive voices like yours, I will try to contact and talk to the authors of the gui projects which are stated above here and arrange an IRC meeting with all interested parties (including anyone who is interested in joining up) to talk a bit. Another advantage of using XUL is that there will be no arguing about the design of the program -grin-. At least not much.

#4

Trees and lists are of utmost importance.

Getting “something simple running like a few buttons” is one hour of work, and most likely in the wrong direction.
If we do this, we need to do it right. Pick an existing standard, strive to implement it fully, and make sure it’s useable, and not just a nice api with a witty name.

#5

Speaking of witty names, how about “jouice”? Java OpenGL User InterfaCE. :wink:

#6

Good, you have ambition. ;D

here is a list of all objects which need to be implemented: http://xulplanet.com/references/elemref/quickref.html

I’m not saying that trees and tables and lists shouldn’t be implemented, on the contrary, just that not everything (iframe anyone?) is applicable to games. And that we should start with the basics.

#7

XUL looks like it’s too big to be implemented in java/opengl without a big development team. It includes fancy things like CSS.
Are there any other standards for this?

As for trees and lists, the wurm gui requires them. :wink:

#8

Yep… I’m interested in a generic GL UI as well. However, I’m going to be playing around with the Java2D/JOGL integration a bit more before looking at anything completely generic. I’ve got a pretty nice custom Java2D GUI API that should port and be extended quite a bit with J2D/JOGL stuff. As you mentioned though I’m quite interested in using alternative ways of configuring the GUI system I’m working on rather than compiled code… XML of course, but I’m interested in using various scripting systems such as Groovy to provide configuration capabilities.

#9

God arguments on having a generic gui system that could work on any platform not only jogl. The xml binding is also important as well as some sort of script to do the basic interface logic but going for xul maybe right away is probably not a good idea as you will spend most of the type implementing stuff you won’t use.

http://xulplanet.com/references/elemref/

I am using something similar for a C++ work im doing, an upgraded version of glgui. This is an old opengl gui system, very simple and compact that suports loading of xml interfaces:

http://glgui.sourceforge.net/

Here the xml schema for the gui xml format. As you can see it’s very simple and easy to implement. Besides it’s works with the concept of shaders to render its gui components:

http://glgui.sourceforge.net/filedocs/index.html

This is something much less bloated than xul and probably a better choice to redesign in Java.

This is infact an abstract gui as it is only necessar to implement two classes each one extending GLUI::IRenderer and GLUI::IUI. All other gui components work on top of these ones.

The GLUI::IRenderer class doesn’t have to be the same as whatever renderer your engine is using. In fact and as a design choice im using several different types of renderers. One of them is a GLUI::DebugRenderer(GLUI:IRenderer) which is a filter that registers render calls in a queue with a timestamp value and then logs them in a file before calling the respective child renderer method.

#10

Just design a simple containment hierarchy with public accessors for the components and use a high level serializer like XStream for the XML-format. Create a root level entry-point (like the browsers “window” or “document” object) and feed this to the context of any existing java scripting language (jython, javascript, beanshell). Don’t waste your time defining XML-schemas and embedded scriptlets.

#11

With that system if classes change then the xml format also changes right ? A schema would ensure that this would not happen. That is I can make changes to the code and to the data model independently. It would only be necessary to rewrite the UIHandler class to load and save xml files.

#12

You can define mappings in XStream, if classes happen to change. The containment hierarchy I described is more like a data model that can be traversed by a renderer/controller similar to a scenegraph, so once defined, changes to that model classes should be minimal. My main point however was, start with a clean, simple and abstract programmable GUI and don’t start with designing the xml-format for the GUI :wink:

#13

Argh! I should have stumbled on this thread much earlier!

I am one of the guys behind FengGUI and I honestly like the discussed ideas. However, it seems that this thread needs a brief update on FengGUIs status. :slight_smile:

That is already done! FengGUI currently supports LWJGL 0.99 and JOGL JSR 231 Beta 02

We want FengGUI to separate between layout (where and how to place Widgets) and appearance. I believe those are different concepts and that it makes sense to separate between them. For the appearance part we have an ANTLR grammar, for the layout part we want to have a XML solution (that allows us to serialize the GUI). The decision about the format for the layout in XML has not been made yet.

The reasons why I designed an ANTLR grammar (in favour of my previously trashed XML solution!) to describe the appearance of Widgets (i.e. Themes) are:

  • While I was building a few example themes with my previous XML format it came to my attention, that those XML files grow extremly large in size and are really cumbersome to handle for the poor artist who is supposed to design the GUI
  • The Theme files are loaded at start-up and then never again. It is not changed and not transmitted over the network.
    There is no obvious reason to have the Theme file portable.
  • I see the advantage of a pre-defined XML schema in the completeness of the specification and not necessarily that it is an XML format (which turn is just another way to structure information). However, since I could not find a well suited XML schema that describes the appearance of Widgets well in the FengGUI environment (FengGUI uses a finite state model that allows to describe different appearances for different, strictly separated states of a Widget), I decided to design an isolated solution.
  • With an ANTLR grammar, you also get a parser out of your grammar, which makes it equally easy to ready in a file.

Ultimately, the human artist who designs the theme is the main actor here, which is why I designed the theme grammar to be human readable, not particularly machine readable :slight_smile:

This is incoporated in our theme grammar as a subset of FengGUIs (quite intuitive) finite state model.

We have a serious lack of documentation, but since the names of the Widgets and the overall tree data structure are pretty much the same like of GUI systems, I think it is easy to get started with FengGUI. However, this is a subjective matter.

Concerning complex (chaotic? :slight_smile: ) gui sctructures I hot-linked thumbnail of a recent FengGUI screenshot. Please click to enlarge.


https://fenggui.dev.java.net/images/screenshot10Thumb.jpg

Leaving the appearance/theme stuff behind, I turn to the XUL standard, which I find and (already found) very promising. I browsed XUL in the beginning during the days when FengGUI was born to get some ideas about how to structure a cool GUI system so that it is not surprising that many of FengGUIs components are aligned to XUL. However I did not follow the standard so that you cannot directly map the XML files on the GUI.

To have a chat session about this topic is definitely a great idea. Unfortunately, I leave on vacation in a couple of days, so that it may happen that I am unavailable (despite from the insane timezone I am living in). But I will try to contribute to this thread.

Concerning FengGUI, we have two commercial stakeholders, which is why I will not abandon the project (besides, I was working on it for almost a year by now in my spare free time). For me personally it would be the best solution (in case we collaborate) to push FengGUI towards XUL step-by-step. But I leave that to discussion. For the time being I invite you to have a look at FengGUI.

It seems that Java-OpenGL-GUI-Libraries are en vogue these days. That is great, because I thought that the currently developed Swing-JOGL bridge will fill the GUI gap (at least for JOGL). However, let us not do the same work over and over again.

Johannes

#14

You make a few interesting points, Schabby. It sure looks promising. I will take a better look at FengGUI.

I wonder if I am able to re-create the Master of Orion 2 main menu with it (including the animations ;D).

http://www.richleader.com/images/Pulse/bargainbin/masteroforion2/master.jpg (link to MOO2 main menu screenshot)

#15

yeah, I think you can (as long as you leave out animated textures, which are scheduled not before spring) :stuck_out_tongue:

It is probably a good idea to add a similar menu to FenGUIs sample box as a proof-of-concept…

Johannes

#16

Yeah, I think so too. One of the reasons I dismissed FenGUI in the beginning… erh… were the screenshots. Not really fair for such a nice project, but I was looking for something specific. I can help with the animated textures if you like. I did some work on animated GIF / MNG / AVI / MPG (although the latter two are a bit overkill). (you need the JMF: http://java.sun.com/products/java-media/jmf/2.1.1/download.html for the webstart demo)

#17

Oh yes, I remember your demo! I first had a hard time to get it working, but it was very impressive in the end!

Your help will be much appreciated! You could help me out of my dilemma with animated textures. I will check out your example more closely as soon as possible. But I really need to turn in now (only four hours left to sleep), so I will come back tomorrow on this thread. N8! :wink:

Johannes

#18

Hi,

I just read the entire thread again in an attempt to sort things a bit in my head. Like the first time, I am excited about the response that was caused by tusakis excellent idea of a “generic” GUI.

Although it may sound obvious to many of you, I want to point out the three levels of the problem.

  • Low Level: The Widget renderer is responsible for actually rendering the GUI. It has to be OpenGL binding idependent and
    the appearance of Widget needs to customizable. The Widgets I am speaking of are buttons, radio buttons, check boxes, lists, trees, tables, etc.
  • Mid Level: A generic interface in form of a XML schema or other grammar is desired to describe the arrangement and the appearance of Widgets. This enables to use the underlying GUI renderer in a data-driven manner while speaking to a
    clear facade at the same time.
  • High Level: The exploiting application describes the GUI system in terms of the XML schema (or other grammar) and passes it
    to the GUI renderer facade.

FengGUI covers pretty much the lowest level. It includes two rendering peers and is capable of rendering almost all commonly used Widget types (well, I should be careful with this statement, some Widgets are not implemented, others miss features…)

FengGUIs ANTLR grammar to describe the appearance of Widgets falls in the Mid Level section. Since it’s naturally required by games to have a very high degree of customization for GUI components, we found it sensible to define our own, powerful grammar. We evaluated the idea of using XML or CSS. While XML (firstly) seemed to be the smartest move (see above) we decided relativlely quick against CSS, because CSS is simply not designed to describe Widgets, rather than HTML elements. For example we haven’t seen any CSS constructs to describe a bevel border on a mouse-over event over sliders, or how large to make the gap between the image and the text on buttons, or to set the delay for animated textures, etc. However, it is possible to map CSS on FengGUI but as a rather high level description language.

I looked into glGUI and XUL this morning and both specs have their charme and disadvantages.

glGUI …

  • …is simple and clean,
  • does only describe the layout, not the appearance (I find that a bonus),
  • is kept to a minimum of constructs
  • incoporates the concept of shaders and renderers (e.g. iconshader, cursorshader)

However, it …

  • …is rather old (from 2003 I think, but that’s not much of a problem i think because GUIs have not changed since then in principle),
  • it seems to miss some aspects, that are not necessary for basic GUIs but might be a requirement for us (e.g. fonts
    with variable character length, tree tables, padding and margin of Widgets
  • The project looks as it is not further maintained (last release is from September 2003), however this point is equivalent to the first item including the same counter argument.

Now, XUL is a much more complete approach to GUIs:

  • It is maintained and has a big supporting community
  • It defines not only the layout
  • but also appearance (CSS, inline HTML)
  • and behaviour (JavaScript)
  • It would enable to run a browser directly on the widget renderer

As the disadvantages (as I seem them)

  • It is bloated
  • The really desired appearance of in-game Widgets can not efficiently designed with CSS
  • Script in scripts for the behaviour is rather messy for our application environment
  • I cannot even estimate the effort of implementing it
  • We would build another colossus among SWT and AWT/Swing. Please keep in mind that we have to work on that
    in our free time. ;D

Which is why I prefer glGUI over XUL (sorry tusaki, don’t get me wrong, no hard feelings). We could start by implementing glGUI and than extend it where it is needed. We could install an advistory board to discuss requested features. People who want another feature added to the standard could turn to the advisory board. By keeping the spec independent from the underlying renderer, enthusiasts are able to implement their own renderer, although I suggest to bundle our forces on a single Widget renderer and the GUI layout/appearance specification.

I would like to suggest to found two groups (maybe even in two separate project workspaces):

  • The specification group that maintains the GUI spec… That is how and where to position Widgets as well as a standarized way to
    describe appearance. I suggest to leave out behaviour (or only cover it roughly and informal) for the sake of simplicity
  • A coder group that implements the specification on the basis of a Widget renderer. It would be great if FengGUI could fill in
    this part.

What do you think of my proposal? Uuuh, gotta go now… I am off to work… I check in this evening again.

Johannes

PS: Like many people pointed out before already, I agree to not see the loading/parsing and saving of of the files as the main issue (although it makes sense to think about whether to the layouting while parsing, or first construct a syntax tree and layout the whole GUI at once by walking this tree).

PSS: Maybe there are more (simple) GUI specs available. Unfortunately I do not have time at the moment to search for them.

#19

The reason why i like glGUI is because of the simplicity and the shader abstraction. A shader is refered in their schema doc by a resource name which must be defined in some xml resouce data block in the resources section. This can be anything from an image to glsl pixel shader program (thinking abstractly not in glGUI of course) or a name of an hardcoded paint method obtained from a theme. Their idea of a renderer also extends beyond the complete independence of JOGL or whatever is used. It would be possible to build a debug renderer that would cache rendering instructions and then flush them at the apropriate time. Their schema also works with the notions of coordinate systems and callbacks. There is no event polling which makes things more clean.

The schema in their site is a good start but like you said it’s old an needs to be revised. My advice is that you use well designed xml schemas for layout and themes. Your idea to separate layout from themes is very nice but don’t bother using anything else besides xml. Yes xml can be hugly sometimes but with a good design this can be minimized. Artists either work with visual development tools or they are very experienced with xml and html so it won’t be a problem to them.

Edit: Theres only one thing i often miss in most gui apis is that all widgtes have a rectangular shape. This is cool for rectangular buttons but it was interesting to have generic widgets which could use any shape for visuals and a proxy shape for mouse hits. It would be also interesting to have undecorated widgets we could decorate with any image shaders or other type of shaders. A sort of flexible widget that could be anything. The master of orion 2 interface would be a chalenge for this type of widget and the use of generic button shapes.

#20

That is by all means something that needs every Widget renderer. I totally agree.

Hmm, the real world has a small obstacle here: As you know, LWJGL has a handy Keyboard and Mouse class that can be used to poll both input devices in question. While the JOGL binding provides callbacks (from AWT/Swing), LWJGL applications need to poll the keyboard and mouse manually. We will need to adopt to that, but it is not a big issue.

I experimented with arbirtrary shapes for Widgets a while ago. It is indeed a nifty thing. In my implemenation, each Widget hold a list of 2D points which formed the outline of the Widget. It was relatively easy to paint the Widget and to draw fancy borders (e.g. letter shaped Widgets, loaded from font data). But when I tried to put them in context, I faced a few (but hard) problems. Most of them apply especially for concave shapes.:

  • can be computational quite expensive to find the Widget that lays below the mouse cursor (well, you mentioned proxy shapes already for that). That is pretty much the problem of determining whether a point lays inside a polygon or not and can be effeciently solved by subdivison, so that this problem would be solvable.
  • layout managers are practically impossible to design (as long as they don’t assume a rectangular bounding box around every Widget, or proxy shapes(?))
  • margins (e.g. paddings) are quite cumbersome to handle
  • proper clipping requires the stencil buffer (or similar) which will render the GUI system useless for systems that need this buffer

Although I also find arbitray shaped Widgets groovy, I believe that they do not belong to the set of most required features. However, we can easily mimic any shape by a rectangular formed Widget with the proper setup So that in the end we haven’t really lost this nice feature. I neglect Widgets having holes because I am sure that we all agree that this would be overkill. :slight_smile:

Yeah… you have a point. Keeping the entire thing consistent in XML is logical. In particular if it comes to cross-references from the layout to the appearance (seldom, but it will happen). Your points about the skilled people using XML are also true of course. Despite from that, DTDs look so nice as de-facto specifications (e.g. in contrast to the rules of the meta-grammar for ANTLR grammars).

I guess I am just not a fan of the XML. I somehow have the impression that it is a bit overrated in our days and often used in places where it is not actually necessary (without saying that it is wrong to use it for a GUI for both appearance and layout)… Maybe in ten years we will all say “remember in those XML-days…” just like years ago when they said that Neural Networks revolutionized AI ;D

Johannes