Smooth scrolling with JOGL/AWT

Hi Ken,

I’ve been building a real-time video editing application
using Java + JOGL, using DirectShow as the codec layer.
This is a huge endeavour for my company and I’ve been
championing Java and OpenGL. Among the many
technical obstacles is smooth rendering synchronised
to the vertical retrace
. Simply put, when I render
a vertical bar scrolling across the screen horizontally,
I expect to see no tearing and no skipped frames,
at full 60Hz. (nothing else is happening: no video stuff is going on)

This C++ SDL example shows that it is possible:
http://olofson.net/examples.html
Look for ‘smoothscroll 1.1’

I also wrote a simple fullscreen app using LWJGL that
does the scrolling bar and it also works smoothly.
There’s the occasional skip frame and it’s likely due
to the GC so I can live with that.

Using JOGL, however, even with AWT GLCanvas
(not the PBuffered GLJPanel), there is no way to
achieve this. I get skip frames almost every second.
The things I’ve tried and seen no improvement:

  1. use Incremental GC
  2. use a P4 CPU with HyperThreading
  3. rendering in AWT thread (1thread=true)
  4. rendering in my own thread (1thread=false)
  5. GLCanvas.setNoAutoRedrawMode(true)
  6. turning on vsync via Windows’s Display Properties
  7. used wglSwapIntervalEXT()

Then I gave LWJGL 0.96 a try since it now supports
AWT. To my disappointment, the same code that
gave smooth results in LWJGL Fullscreen now also
skips frames in AWT mode.

So my question now is, is there anyway to achieve
what the C++ SDL example did, while having
OpenGL work inside AWT? Is there anyway to
get absolute full control over OpenGL and all the
swap buffer calls as an AWT component yet have
no dependence on the AWT dispatch thread?
I’m willing to do just about anything to get this
working, because otherwise the Java+JOGL route
is a no-go for my job.

Please advise, thanks!

Just curious, why would you need to scroll? Couldn’t you just move the camera position to simulate scrolling?

You missed the point: the scrolling bar is just
an acid test for vertical retrace synchronisation.

Do you have a test case which shows tearing? I recall looking into a bug a while ago that reported problems with swapBuffers and the test case was pretty convoluted and actually buggy as far as I could tell. We need to expose vertical sync functionality portably in JOGL and you can feel free to file an RFE about that; I’ll try to get it into the next beta. I am quite sure that it should be possible to do that SDL smoothscroll demo in Java with JOGL.

Hi Ken,

Yes I’ve just found the root of the problem.
First use the attached source below:


` package experiments;

import java.awt.BorderLayout;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.awt.event.KeyEvent;
import java.awt.event.KeyListener;

import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.SwingUtilities;

import net.java.games.jogl.Animator;
import net.java.games.jogl.GL;
import net.java.games.jogl.GLCanvas;
import net.java.games.jogl.GLCapabilities;
import net.java.games.jogl.GLDrawable;
import net.java.games.jogl.GLDrawableFactory;
import net.java.games.jogl.GLEventListener;
import net.java.games.jogl.GLU;

public class SmoothScrollJOGL2
extends JFrame
implements KeyListener, GLEventListener
{
GraphicsDevice device;

  GLCanvas canvas;
  Animator animator;
  
  JLabel label = new JLabel("hahaha");
  
  long   elapsedStart;
  double elapsedSecond;
  
  /**
   * Constructor.
   *
   */
  public SmoothScrollJOGL2()
  {
        device = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();

      canvas = GLDrawableFactory.getFactory().createGLCanvas(new GLCapabilities());
        canvas.addGLEventListener(this);
        
        setUndecorated(true);

        getContentPane().setLayout(new BorderLayout());
        getContentPane().add(canvas, BorderLayout.CENTER);
        getContentPane().add(label, BorderLayout.SOUTH);
        
      animator = new Animator(canvas);

        device.setFullScreenWindow(this);

        elapsedStart = System.nanoTime();
        
      animator.start();

        addKeyListener(this);
  }
  
  public void keyTyped(KeyEvent e) {}
  public void keyPressed(KeyEvent e) {}

  public void keyReleased(KeyEvent e)
  {
    new Thread(new Runnable() 
               {
                          public void run() 
                                {
                            animator.stop();
                            System.exit(0);
                                }
                       }).start();
  }


  public void displayChanged(GLDrawable drawable, boolean modeChanged, boolean deviceChanged) {}
  public void init(GLDrawable drawable) {}
  
  public void reshape(GLDrawable drawable, int x, int y, int width, int height)
  {
        GL  gl  = drawable.getGL();
        GLU glu = drawable.getGLU();

        // turn on vsync, just to be sure
        gl.wglSwapIntervalEXT(1);
        
    gl.glMatrixMode(GL.GL_PROJECTION);
    gl.glLoadIdentity();

        // choose major depending on which axis is bigger
        if(width > height)
        {
              float AR = (float)height / width; // width-major

              gl.glOrtho(-1f, +1f,  // left, right
                             -AR, +AR,  // bottom, top
                             -1f, +1f); // near, far
        }
        else
        {
              float AR = (float)width  / height; // height-major
        
              gl.glOrtho(-AR, +AR,  // left, right
                             -1f, +1f,  // bottom, top
                             -1f, +1f); // near, far
        }

        System.out.println("reshape(): " + width + "," + height);
  }

  public void display(GLDrawable drawable)
  {
        elapsedSecond = (System.nanoTime() - elapsedStart) / 1000000000.0;
        
        GL  gl  = drawable.getGL();
        GLU glu = drawable.getGLU();


        gl.glClear(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT);

        
        gl.glColor3f(1,1,1);
        
    gl.glMatrixMode(GL.GL_PROJECTION);
    gl.glLoadIdentity();
        gl.glTranslatef((float)Math.sin(elapsedSecond), 0, 0);
        
        
    float x = 0.1f / 2;
    float y = 2 / 2;

    gl.glBegin(GL.GL_QUADS);
        
      gl.glVertex3f(-x, -y, 0); // BL
      gl.glVertex3f(+x, -y, 0); // BR
      gl.glVertex3f(+x, +y, 0); // TR
      gl.glVertex3f(-x, +y, 0); // TL
              
        gl.glEnd();

        SwingUtilities.invokeLater(new Runnable() { public void run() { label.setText(""+elapsedSecond); } });
  }


  public static void main(String[] args)
  {
        new SmoothScrollJOGL2();
  }

}

`


If you run this in “jogl.1thread=true” mode
you will notice very bad skipping frames.
When in “jogl.1thread=false” mode it will be
smoother but still skip frames regularly.

Now add the following code snippet at the
start of net.java.games.jogl.GLCanvas.reshape():


` public void reshape(int x, int y, int width, int height) { if(width == getWidth() && height == getHeight() && x == getX() && y == getY()) { return; } `

Now run the test app in “jogl.1thread=false”
mode and you should see no more skipped
frames. If “jogl.1thread=true” then there will
still be occasional skip frames. I’ve tested this
on a nVidia 6600GT with latest drivers (71.84)
and on a mobile ATI X700 with bundled drivers
and it works. I’ve also collected timing data
and plotted Excel charts to show the stability
of the frames.

This usage scenario comes up when using JOGL
in any type of editor-type app with real-time
previewing and the skip frames is really quite
unacceptable. I think the way to solve it is
not to make the rendering happen on the
AWT thread but on a user-defined thread.

.rex

BTW, can some of the kind members of this
forum try out the above test app and let us
know of your results? Thanks…

The updating of the label is what is causing the constant reshape() calls on your Canvas. If you have soft real-time requirements you should probably be using OpenGL to draw your text. Try taking that out and see if it improves the situation. You should also be running with -Dsun.java2d.noddraw=true on Windows if you aren’t already, as that switch changes the behavior of the full-screen APIs.

Ken,

But I have hard real-time requirements
and the application I’m building requires
real-time updates of Swing widgets in
an IDE-type environment. This is not your
usual game UI where everything can be
rendered in a single OpenGL window.
It’s more along the lines of Maya and
Apple’s Motion.

.rex

How much of this issue is the use of JLabel?

I had related issues with JLabel, and concluded that it wasn’t really intended to be used for text that was being changed (indeed as its name implies). I moved to using a JTextField on which I had called ‘setEditable(false)’ as can be seen in my webstart demo here:

http://abraded.co.uk/jogl/

By the way, I don’t see the tearing with your demo if I run with -Dsun.java2d.noddraw=true as Ken suggests. The tearing is very bad without this.

I can’t tell however if the above helps: basically for me running your testcase both JLabel and JTextField get tearing without the -D flag, and neither get tearing with the -D flag, but at least the ‘reshape’ doesn’t get called with JTextField.

Another major issue I had mixing Swing with JOGL is that updates to Swing components can be expensive enough to cause you to miss frames in the GLCanvas if you are trying to hit the vertical refresh rate. I minimised this by minimising my updates to the Swing components - primarily by ensuring I was not updating the Swing components every frame.

Rob

Same here. High rate update of JLabels (pointing device tracking for instance) generated fickering problems and JTextField solved the issue.

Hi Rob,

One of the problems now is that GLCanvas.reshape()
doesn’t not check if the passed parameters are the
same as current values already, resulting in every call
to reshape() being executed all the way. By placing the
check and doing an early-out, I noticed significant
improvements in speed, even when it is GL is running
on the AWT thread.

Next, windowed mode operation is much more
important to me than fullscreen mode, because I’m
developing an editor, not a fullscreen game.

I also update my Swing components minimally. However,
the real problem now is that JOGL does not work
well with Swing components being updated in real-time when:

[]in windowed mode
[
]when running on the AWT thread with jogl.1thread=true
[*]when there’s no early-out in GLCanvas.reshape()

I’ve demonstrated the problem and provided a solution.
It may be a naive solution because I don’t know if it
breaks anything, but at least it is a starting point for the
JOGL team to consider. Otherwise we cannot invest a
million dollars (I’m serious) developing our product using
this Java+JOGL approach.

.rex

Hi Ken,

[quote]You should also be running with -Dsun.java2d.noddraw=true on Windows if you aren’t already, as that switch changes the behavior of the full-screen APIs.
[/quote]
My main concern is windowed-mode operation,
as I’m developing an editor, not a fullscreen game.
If I use ‘noddraw=true’, does that mean I loose
DirectDraw acceleration in Swing? That won’t do
because I have a lot of Swing updates to do in
real-time.

Do you know what’s the status of Java2D’s OpenGL
acceleration in JDK6? Chris mentioned that they’ve
implemented single-thread+queued pipeline ala
Swing, presumably using a private GL context.
Does it work well with JOGL now? I certainly do not
wish to be switching GL contexts (between Java2D
and JOGL) 30 times per second.

.rex

Rex,

[quote] One of the problems now is that GLCanvas.reshape()
doesn’t not check …
[/quote]
My point was that reshape should not be being called with the frequency that it is. If you use a JTextField instead of a JLabel in your posted demo code, the reshape method only gets called once in a run of the test, not all the time. Your change to reshape() in GLCanvas is a work-around to an issue that is only occuring, I think, because of your (mis) use of JLabel.

Rob

Rob,

I used JLabel intentionally, knowing that
it causes reshape() calls, to show the fact
that GLCanvas.reshape() should have an
early-out check to protect performance
against such things. Surely we cannot
expect everyone to know about all these
little things like updating a JLabel causing
reshape() calls. A general-purpose and
performance-sensitive API like JOGL should
be coded defensively. Imagine some uninformed
dude comes along and writes a widget plug-in
for my editor using JLabels and suddenly
the OpenGL rendering starts to stutter.
Would you like to debug something like that?
So I think that’s unacceptable. There are so
many pitfalls in Swing that many of us have
grown used to working around them but I
really don’t think that’s what ease-of-use means.

Now if you extend the JLabel example to
something more ‘real-world’, like a JPanel doing
some re-layout in another split pane, which
ultimately doesn’t change the size of the GLCanvas:
should it add stutters to an otherwise perfectly
steady 60Hz rendering of a preview?

.rex

On my machine it makes no difference whether the GLCanvas skips reshape notifications or whether a JTextField is used in place of the JLabel. As long as the Swing component is being updated the main frame exhibits jitter. Moving the Swing component to its own heavyweight JFrame seems to slightly reduce the jitter in the steady-state of the application though it still jitters at least initially. For this reason I don’t think it’s a good idea to add heuristics to GLCanvas.reshape() to skip notifications.

I think you should look at Apple’s Final Cut Pro as an example of a similar application to yours. I used it a couple of years ago for a couple of video editing jobs. It has a preview mode where it can render the final video to an external TV screen. As far as I remember, while it was performing that rendering, there was no activity on the computer screen aside from the option of pressing ‘escape’ to stop the preview. I think you should design the modes of your application to separate out the real-time portions from the interactive portions and make sure you can isolate your app from almost everything else going on in the system, including the widget library, while performing real-time work. I think you would need to do this no matter what third-party widget library you chose. Of course, if you have complete control over the widgets then you can probably play more games with their performance and integration with the application.

Yes, -Dsun.java2d.noddraw=true disables DirectDraw acceleration for Swing widgets on Windows and falls back to Java2D’s GDI pipeline. DirectDraw and OpenGL are fundamentally incompatible at the driver level and I have seen problems where they are used in separate windows in the same application (like for the Java Web Start console). Java2D’s OpenGL pipeline is promising, but the kind of integration you mention, where both JOGL and Java2D share the same OpenGL context, is not a trivial result of Java2D switching to OpenGL. More integration work is needed to allow JOGL or any third-party Java binding to OpenGL to interoperate with Java2D at this level. With some luck some of this may show up in Mustang but given that this is still quite a ways off you should plan to build your app with the tools available today if you’re going to use Java and JOGL.

[quote]Ken,

But I have hard real-time requirements
and the application I’m building requires
real-time updates of Swing widgets in
an IDE-type environment. This is not your
usual game UI where everything can be
rendered in a single OpenGL window.
It’s more along the lines of Maya and
Apple’s Motion.

.rex
[/quote]
I might be off the mark here but if you have hard real time requirements is Java/JOGL really suitable? What are you doing about the garbage collector for instance? It will be exceptionally hard to ensure the real time properties of the system across different JVMs and platforms (especially using Swing, since its speed depends a lot on the platform on which it is running).

[quote]I might be off the mark here but if you have hard real time requirements is Java/JOGL really suitable? What are you doing about the garbage collector for instance? It will be exceptionally hard to ensure the real time properties of the system across different JVMs and platforms (especially using Swing, since its speed depends a lot on the platform on which it is running).
[/quote]
Well, the project has met several milestones already
and it looks like Java/JOGL is definitely up to the job.
The main issue of this topic boils down to AWT/DirectDraw
interfering with JOGL.

As for the GC, so far it’s not a problem since it can
be tuned and we have HT or dual-processor hardware
requirements.

.rex

Ken,

Thanks for trying out the demo code.

[quote]On my machine it makes no difference whether the GLCanvas skips reshape notifications or whether a JTextField is used in place of the JLabel. As long as the Swing component is being updated the main frame exhibits jitter.
[/quote]
Have you tried running it with jogl.1thread=false?
On my WinXP, P4HT and nVidia 6600GT setup,
it gives almost no jitters for extended periods of
time.

[quote]Moving the Swing component to its own heavyweight JFrame seems to slightly reduce the jitter in the steady-state of the application though it still jitters at least initially.
[/quote]
Yes that’s the same as what I see.

[quote]For this reason I don’t think it’s a good idea to add heuristics to GLCanvas.reshape() to skip notifications.
[/quote]
Actually I don’t quite understand why.
I’ve been using the modified GLCanvas
for a few days within a Swing-based GUI
with docking windows (Infonode Docking Windows)
and it seems to work fine so far. I think
the jitter is caused by the call super.reshape(…)
to Canvas itself.

[quote]external TV screen. As far as I remember, while it was performing that rendering, there was no activity on the computer screen aside from the option of pressing ‘escape’ to stop the preview. I think you should design the modes of your application to separate out the real-time portions from the interactive portions and make sure you can isolate your app from almost everything else going on in the system, including the widget library, while performing real-time work.
[/quote]
Yes I certainly do not expect jitter-free
rendering when there’s interaction going
on. However, the issue here is I can’t even
update some small widgets to display stats
(e.g. timecode, framerate and CPU usage)
without causing jitters.

[quote]Java2D’s OpenGL pipeline is promising, but the kind of integration you mention, where both JOGL and Java2D share the same OpenGL context, is not a trivial result of Java2D switching to OpenGL. More integration work is needed to allow JOGL or any third-party Java binding to OpenGL to interoperate with Java2D at this level.
[/quote]
But at least it is possible to not rely on
DirectDraw so the driver-level conflicts as
you said can be avoided. And this alone is
good news to me.

.rex

[quote]Have you tried running it with jogl.1thread=false?
On my WinXP, P4HT and nVidia 6600GT setup,
it gives almost no jitters for extended periods of
time.
[/quote]
I just tried it again with that flag and it still jitters during every traversal of the bar left and right.

[quote]I’ve been using the modified GLCanvas
for a few days within a Swing-based GUI
with docking windows (Infonode Docking Windows)
and it seems to work fine so far. I think
the jitter is caused by the call super.reshape(…)
to Canvas itself.
[/quote]
OK, I didn’t realize you wanted to completely short-circuit the reshape process; I had just been avoiding the call to the GLEventListener’s reshape method. I just tried modifying GLCanvas in the way you suggest and while the jitter seems a little better it is still there and occurs on every sweep of the bar left and right on my machine.

[quote]Yes I certainly do not expect jitter-free
rendering when there’s interaction going
on. However, the issue here is I can’t even
update some small widgets to display stats
(e.g. timecode, framerate and CPU usage)
without causing jitters.
[/quote]
I really think you should be rendering these things as text within your OpenGL display. Take a look at the control panel in the JCanyon demo, for example; it’s rendered using the GLUT bitmaps.

Regarding the OpenGL pipeline:

[quote]But at least it is possible to not rely on
DirectDraw so the driver-level conflicts as
you said can be avoided. And this alone is
good news to me.
[/quote]
You can already avoid using DirectDraw on Windows with -Dsun.java2d.noddraw=true; it falls back to the GDI rendering path. I haven’t seen any performance problems with any JOGL apps with this flag enabled, and it should be enabled for all JOGL apps on Windows. However, turning on Java2D’s OpenGL pipeline currently causes significant interference with JOGL, apparently because both are trying to use the same underlying mechanisms to draw to the screen. Hopefully we can figure out some of these issues and also achieve better integration with Java2D in the 1.6 timeframe.

No, it is quite possible. We’ve been building apps that scroll real-time spectrogram data (1000 fps) over extended periods of times (days) using Java and OpenGL on a normal workstation. The GC problem is overcome by using pooled resources and object re-use. And of course a good profiler and some tuned programming.

I think for the most part this is where the power of the Java OpenGL cooperation lies - you get the power of OpenGL with the versatility, ease-of-use and platform independence of Java.

Rex makes a valid point, I am also championing JOGL as the new alternative to GL4Java at my company, but I am currently having a lot of problems with the threading model in JOGL. It turns out that its not quite that simple to convert from GL4Java to JOGL.