java2d-pipeline / glReadPixels / alpha-component

[Saxer]

Hello,

As mentioned in several posts before I’m experimenting with the jogl-java2d-pipeline.

Right now I have again a problem with the glReadPixel-Method. I want to determine which pixels are transparent. So I just look at the 4th pixel-component to see if it’s != 0. With an enabled pipeline every pixel has an alpha-value of 255. What I know from the technical stuff behind the pipeline is, that opengl renders directly into the swingbuffer. Am I right with the suggestion, that I’m reading the content of the - lets say - jframe and not just my gljpanel?

My solution works great without the pipeline enabled.

The code around the readpixels:

mouseBuffer = BufferUtil.newByteBuffer((renderPanel.getWidth())
					* (renderPanel.getHeight()) * 4);
gl.glPixelStorei(GL.GL_PACK_ALIGNMENT, 1);
gl.glReadPixels(0, 0, renderPanel.getWidth(), renderPanel	.getHeight(), GL.GL_RGBA, GL.GL_UNSIGNED_BYTE, mouseBuffer);

The code which determins the alpha-value:

int index = (mouseBuffer.get(pos + 3) & 0xFF);
return index != 0;

The calculation of pos is ok because it works for other purposes.

The buffer contains more than I would expect. Is there a way to simple extract the real 3d-rendered-content? I could draw the scene into a bufferedimage and check the pixels, but I don’t think it would help the performance of my app

Thanks for any comments / help
Greets Klemens

[broumbroum]

I don’t know about 2d pipeline in jogl. But I know that it can be enabled for Java2d app like this way: (-J)-Dsun.java2d.opengl=True as a command line arg to java.
Why do you pay time to Java2D when your goal is 3D scening? That’d be the question.

[Saxer]

Well, I’m building swing-components with 3D features. Therefore I need this. I know how to enable the pipeline… the described problem only exists with enabled pipeline.

[kbr]

Yes, you should be able to see the entire Swing back buffer and not just your GLJPanel’s contents if you’re careful. Note that the OpenGL viewport and scissor region are set up to surround just the GLJPanel’s region before your GLEventListener is called, so you’ll need to change them if you want to read back more than just your GLJPanel.

Aside from that I don’t know why you’re seeing alpha=255 pixels everywhere. If you could please provide a small and clear test case showing the problem we can look into it.

You may also find it useful to look at the source code of the XTrans demo in the jogl-demos workspace, which does some 3D effects with Swing components using the Java 2D/JOGL bridge.

[Saxer]

Yes I’ll provide one test-case later this afternoon. The reason of the 255’s seems to be, that I read the final composed pixels of the buffer which are fully opaque.

[Saxer]

Here’s my testcase I hope it shows the problem. I cant test it with the pipeline at this maschine. Just click on the quad in the middle and watch at the console.

import java.awt.BorderLayout;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import java.nio.ByteBuffer;

import javax.media.opengl.GL;
import javax.media.opengl.GLAutoDrawable;
import javax.media.opengl.GLCapabilities;
import javax.media.opengl.GLEventListener;
import javax.media.opengl.GLJPanel;
import javax.media.opengl.glu.GLU;
import javax.swing.JButton;
import javax.swing.JFrame;

import com.sun.opengl.util.BufferUtil;

public class PipelineError {

	private static ByteBuffer mouseBuffer;
	/**
	 * @param args
	 */
	public static void main(String[] args) {
		
		GLCapabilities caps = new GLCapabilities();
		caps.setAlphaBits(8);
		
		final GLJPanel panel = new GLJPanel(caps);
		panel.setOpaque(false);
		panel.addGLEventListener(new GLEventListener() {

			public void display(GLAutoDrawable arg0) {
				GL gl = arg0.getGL();
				if (panel.shouldPreserveColorBufferIfTranslucent()) {
			         gl.glClear(GL.GL_DEPTH_BUFFER_BIT);
			     } else {
			         gl.glClear(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT);
			     }

				gl.glEnable(GL.GL_DEPTH_TEST);
				gl.glColor3f(0.6f, 0.6f,0.6f);
				
				gl.glColor3f(1f, 0f, 1f);
				gl.glBegin(GL.GL_QUADS);
				gl.glVertex3f(-1, 1, 0);
				gl.glVertex3f(-1, -1, 0);
				gl.glVertex3f(1, -1, 0);
				gl.glVertex3f(1, 1, 0);
				gl.glEnd();
				
				mouseBuffer = BufferUtil.newByteBuffer((panel.getWidth())
						* (panel.getHeight()) * 4);
				gl.glPixelStorei(GL.GL_PACK_ALIGNMENT, 1);
				gl.glReadPixels(0, 0, panel.getWidth(), panel.getHeight(), GL.GL_RGBA, GL.GL_UNSIGNED_BYTE, mouseBuffer);
			}

			public void displayChanged(GLAutoDrawable arg0, boolean arg1, boolean arg2) {
				
			}

			public void init(GLAutoDrawable arg0) {
				arg0.getGL().glClearColor(1,0,1,0);
	            int clearBits = GL.GL_DEPTH_BUFFER_BIT;
	            clearBits |= GL.GL_COLOR_BUFFER_BIT;
	            
	            arg0.getGL().glClear(clearBits);
			}

			public void reshape(GLAutoDrawable arg0, int x, int y, int width, int height) {
				GL gl = arg0.getGL();
				gl.glViewport(x, y, width, height);
		        gl.glMatrixMode(GL.GL_PROJECTION);
		        gl.glLoadIdentity();
		        new GLU().gluPerspective(45.0, (float) width / height, 1.0, 400.0);
		        
		        gl.glMatrixMode(GL.GL_MODELVIEW);
		        gl.glLoadIdentity();
		        
		        new GLU().gluLookAt(
		                0,0,5, 0,0,0, 
		                0, 1, 0);

			}
			
		});
		
		panel.addMouseListener(new MouseAdapter() {
			@Override
			public void mousePressed(MouseEvent e) {
				
				int x = e.getX();
				int y = e.getY();
				
				if (mouseBuffer == null || x < 0 || y < 0) {
					return;
				}

				int bufX = x;
				int bufY = panel.getHeight() - y;

				int pos = (bufY * panel.getWidth() + bufX) * 4;

				if (pos >= 0 && pos < mouseBuffer.capacity()) {
					int index = (mouseBuffer.get(pos + 3) & 0xFF);
					System.out.println(index);
				}
			}
		});
		
		
		JFrame frame = new JFrame();
		frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		frame.setSize(400, 400);
		frame.setGlassPane(panel);
		panel.setVisible(true);
		frame.add(new JButton("test-Button"), BorderLayout.CENTER);
		
		frame.setVisible(true);
	}

}

[campbell]

Here’s the deal. Swing’s backbuffer is always opaque. There is currently no way to specify that you’d like Swing to allocate an alpha channel with its backbuffer. When you have the OpenGL-based Java 2D pipeline enabled, and you’re using GLJPanel, Swing’s backbuffer is allocated as a pbuffer or FBO without an alpha channel. It is therefore expected that you will see all opaque alpha values if you read back directly from the Swing backbuffer using glReadPixels().

When the OpenGL-based Java 2D pipeline is not enabled, and you’re using GLJPanel, JOGL will allocate a pbuffer that does contain an alpha channel, which is necessary for properly compositing the contents of that offscreen pbuffer onto the actual Swing backbuffer. So in that case, you’re using glReadPixels() to read back from that alpha channel enabled pbuffer, and therefore some of the pixels may appear to be transparent.

Chris

[Saxer]

I expected something like that.

How could I archieve the functionality which is described in the demo? Rendering into an additional pbuffer? Or rendering into an BufferedImage (not sure if this would work). The mentioned functionality is completely essential for my projekt :-(. I don’t need it in every frame just in few - lets call them - key-frames.

It would be enough to simply know if pixel x,y belongs to the gljpanel or not. (I use this technique to modify the contains(x,y) method in order to hide the gljpanel from mouseevents)
Maybe an bad idea:
readPixels => Buffer1
doRendering
readPixels => Buffer2

if (Buffer1(x,y) != Buffer2(x,y)) {
somsething happend in the rendering => rendered-alpha != 255
}

That wouldn’t cover a rendered blue pixel ontop of a blue pixel which is already there. Should the backbuffer be lets say completely black / white in all the regions which are not affected by the rendering?

I hope you’ve got an positive answer :-\

[Saxer]

No suggestions? I’m quite stuck with this.

I tried to set the clearColor to opaque-black and take black as the decision color instead of the alpha value. That results in black rectangles (the gljpanel) even if I clear the scene after the glReadPixels call with a translucent color.

I don’t need the readpixels for presentation - just for background-functionality. Therefore it could be useful to render into the mousebuffer directly (and with an alpha component). Unfortunately I don’t know how to do this exactly. To render within a different context (pbuffer or something) seems to be possible but very difficult, cause I would have to setup all of the transforms and textures for that context.

thanks in advance
Klemens

[kbr]

Look at the source for the XTrans demo in the jogl-demos workspace. If you are able to manually trigger a repaint in order to create your readback (“mouse”) buffer, then you can cause Swing to render itself into a BufferedImage (create a BufferedImage of the appropriate size, call getGraphics() on it, and pass that into the paint(Graphics) method of your top-level component). You could potentially then use that BufferedImage for your queries while still doing your on-screen rendering another way. The XTrans demo does something vaguely similar to this except it takes advantage of the Java 2D / JOGL bridge to interact with VolatileImages and it doesn’t do any readback of this sort except for rendering.

[Saxer]

I checked the XTrans-Demo. It seems, you don’t use a GLJPanel within this demo. I manage the rendering stuff of the nested components as you do (or even quite similar).

The only hint I could eventuelly use is this isOptimizedDrawingEnabled() method. But I don’t know if this would help (Thinking about doing setting it to false when I screenshot the GLJPanel (see below).

The problem shows up, when I paint the GLJPanel into a dedicated Graphics-object (the one I got from the BufferdImage).

With disabled pipeline everything works as it should (BufferedImage setup in pain-method an super.paint(image.getGraphics)). I get translucent pixels with correct alpha-values to work with. Wit enabed pipeline I get an fully opaque image. I tried every possible glClear-Setting (color-bit only, color + depth, depth only, nothing at all). Only with no glClear-call I get transparent pixels (unfortunately the content is totally disorderd with some bad artefacts - as you might now).

I know it is against the intention of the pipeline, but is there a way to workaround the “render to the swingbuffer”-thing in certain situations. I don’t know how you do it exactly, but is it possible to determine if the passed graphics-object comes from an image? And for that case disable the backbuffer-thing.

Maybe an public BufferedImage getContentAsImage() method would be the nicest feature ;D

I’m not sure if the Screenshot-class which shippes with jogl would solve the problem. I’ll give it a try soon.

I forgot: I didn’t try it with latest jogl - lastest jdk 6 update. I 'll do it this afternoon.

[kbr]

If the GLJPanel isn’t working for you due to the Java 2D / JOGL bridge then you can build your own. You can use either a GLPbuffer or, better, a Frame Buffer Object to render your 3D scene and then use the JOGL-internal Java2D class in similar fashion to how the GLJPanel does in order to render that FBO as a texture to the Java 2D / Swing back buffer.

[Saxer]

Hmmm… I’ll have a look into the GLJPanel implementation to see if I can insert my identifikation-stuff somewhere within it’s rendering cycle. Otherwise I’ll have to do one additional rendering-loop with some kind of color-masking (every geometry would be paintet in black) to determine which pixels are “inside” and which are “outside”.

[Saxer]

Well as always I run from one sloved problem into the next. Right now I use the depthbuffer to determine which pixels have been drawn and which not.

I’m not quite sure which dimension my readback buffer should have cause of the viewport-thing of the java2d-pipeline:

Should I do this?

int width = renderPanel.getWidth() + viewPort.x;
int height = renderPanel.getHeight() + viewPort.y;
containsBufferFloat = BufferUtil.newFloatBuffer(width * height);
gl.glPixelStorei(GL.GL_PACK_ALIGNMENT, 1);
gl.glReadPixels(0, 0, width, height, GL.GL_DEPTH_COMPONENT, GL.GL_FLOAT, containsBufferFloat);

or that?

int width = renderPanel.getWidth();
int height = renderPanel.getHeight();
containsBufferFloat = BufferUtil.newFloatBuffer(width * height);
gl.glPixelStorei(GL.GL_PACK_ALIGNMENT, 1);
gl.glReadPixels(0, 0, width, height, GL.GL_DEPTH_COMPONENT, GL.GL_FLOAT, containsBufferFloat);

Of course I have to deal with the “translation” when I convert the mouse-position to find the right position… I tried the first-variant cause I felt, that just reading “renderpanel.getWidth”-pixels in a row would give me not all the pixels of the rendering (because the viewport begins not in 0,0)

Well nevertheless it’s no big problem and could be solved by simply checking out what information is returned.

Unfortunately the depth-buffer seems to be filled wrong (I don’t know if this i a graphics-vendor bug). I have pictures wihtin I tried to visualise the contents of the buffer (1.0 means blue - 0.0 means black) . In my case I use 2 GLJPanels within a LayeredPane - otherwise (with only one panel) the content of the buffer seems to be ok. These pictures visualise the depthbuffers of the two panels (nothing was rendered so they should be completely black).

http://www.fusion-laboratory.de/zBuffer/test1.png

http://www.fusion-laboratory.de/zBuffer/test2.png

(note that the pixels in the top-left-corner seem to be identical)

I completely don’t understand this behaviour

[lhkbob]

Make sure you clear the depth buffer bit, too, just like you would for clearing the color bit, but pass in GL.GL_DEPTH_BUFFER_BIT. Also, for better performance, you can “or” all of the bits together in glClear() cal.

[kbr]

What are you really trying to read back? The GLJPanel’s contents or the entire Swing back buffer?

You should be calling approximately

int width = renderPanel.getWidth();
int height = renderPanel.getHeight();
containsBufferFloat = BufferUtil.newFloatBuffer(width * height);
gl.glPixelStorei(GL.GL_PACK_ALIGNMENT, 1);
gl.glReadPixels(viewPort.x, viewPort.y, width, height, GL.GL_DEPTH_COMPONENT, GL.GL_FLOAT, containsBufferFloat);

[Saxer]

you’re right with your code. The first part of my post was more like a philosophical question.

The problem starts with the actual depth-values. It seems, that those values are not cleared correctly. In my “vision” glClear should “reset” all of the values to 1.0 (or whatever one has configured). Without enabled pipeline my result are as expected (totally black “images”).

[kbr]

Yes, it should. In this area, the main difference between having the Java 2D / OpenGL pipeline enabled and disabled is that when it’s disabled, the (x, y) offset to your component is always (0, 0), and when it’s enabled, the (x, y) offset to your component is its (x, y) offset on the actual back buffer. You may need to use some of the internal methods on the Java2D class to compute the region on the Swing back buffer in OpenGL coordinates of your panel. I would also suggest you try reading back the color buffer first instead of the depth buffer so it’s easier for you to calibrate your code and figure out what’s going on. At that point you can switch over to reading the depth buffer, which should then work 100%.

[Saxer]

Well… the viewport-thing was exactly the reason of all the trouble. Although I can’t figure out, why it was working “partially” before. (The “blue” artifacts appeared just in particular positions.)

Thanks for your help. Now the performance on “slow” systems (with pipeline-support) could be increased conspicuously .

[kbr]

Glad things are working.