Offscreen Canvas3D Update for hand-made Occlusion

Engines, Libraries and Tools Java 3D
#1

Hello everybody.

Didn’t find a solution searching here or in google. Perhaps someone can forward me to some hints or help me out a little bit. I’m just starting off with Java3D. Normal display and manipulation of my universe works fine, but now I’m getting into tricky parts…

My overall goal is to display java3D-objects above an image or video, where the 3d stuff is occluded by real-life-objects out of the (video)image. Of course I don’t get the 3d-info out of my image, but I rather have a prepared mask for occlusion or representative 3d-models for that (which I don’t want to draw but only use for occlusion).

Unfortunatelly I didn’t find out how it would be possible to use 3d-models only for filling the z-Buffer, but not the display… (java3D seems to be TOO high-level for that).
Any idea?

My other plan is to grab the canvas3D-image as a BufferedImage, do an AND-operation with my mask-image and underlay the result with the background-image. This works fine for the first frame, but (and here I get to the point) if I have any Behaviours in my world changing positions for example the Canvas3D does not update itself… and my grabbed image of the 3D-world is always the first one…
Functions like renderOffScreenBuffer() or startRenderer() don’t seem to change a thing…

Any advice how to still get a 3D-world in a Canvas3D to run, which is NOT displayed directly (only drawn offscreen)?

Here some sourcecode:

public class Something extends Frame implements KeyListener
{

BufferedImage bImage;
...

setLayout(new FlowLayout());
GraphicsConfiguration config = SimpleUniverse.getPreferredConfiguration();

// initialise canvas OFFSCREEN (flag = true)
canvas3D = new Canvas3D(config, true);
canvas3D.getScreen3D().setSize(350, 350);
canvas3D.getScreen3D().setPhysicalScreenWidth(0.254 / 90.0 * xsize);
canvas3D.getScreen3D().setPhysicalScreenHeight(0.254 / 90.0 * ysize);

...

// creates some objects and Behaviours which don't need user interaction
BranchGroup scene = createSceneGraph();
SimpleUniverse simpleU = new SimpleUniverse(canvas3D);
simpleU.getViewingPlatform().setNominalViewingTransform();
simpleU.addBranchGraph(scene);

...

bImage = new BufferedImage(350, 350, BufferedImage.TYPE_4BYTE_ABGR);
ImageComponent2D buffer = new ImageComponent2D(ImageComponent.FORMAT_RGBA, bImage);
canvas3D.setOffScreenBuffer(buffer);
canvas3D.renderOffScreenBuffer();
canvas3D.waitForOffScreenRendering();
bImage = canvas3D.getOffScreenBuffer().getImage();

// I add the buffered image to some panel, afterwards I see the first frame of the world in it
// ImagePanel extend Canvas
ImagePanel panel = new ImagePanel(bImage);
add(panel);

...

// after this I can see the Canvas in my Frame displaying the BufferedImage-copy of the 3dworld... but only the first frame

// I defined a keyListener to my Frame which triggers this:

public void update3DView()
{
      canvas3D.startRenderer();
      canvas3D.renderOffScreenBuffer();
      canvas3D.waitForOffScreenRendering();
      
      bImage = canvas3D.getOffScreenBuffer().getImage();
      
      // whatever I try it does not trigger an update
      panel.repaint();
      canvas3D.repaint();
      repaint();
}

Well. That’s it so far.
Don’t know where the error could lie…
some missing calls for the canvas3D/theUniverse, some missing AWT update, … hmm.

If I use the same universe created by createSceneGraph() with an onscreen Canvas3D all animation is triggered perfectly.

Thanks in advance,
cu,

Konterfei.

#2

As i udnerstand it, what yoi uare trying to do is a matte.

You REALLY want the stencil buffer for this. Unfortunately Java3D doesnt have stencil buffer support yet… its coming.

Edit: Just talked to Kevin on the J3D team. He suggests you loo at a “Depth Only Raster.”

He thinks you can use it t do your mattes, but he says it hasnt got a lot of use so if you have problems, let us know!

#3

Thanks so far. I’ll give it a try. Have no clue how to use it yet, but I’ll see… It’s “javax.media.j3d.Raster” you are suggesting, right?

[quote]but he says it hasnt got a lot of use
[/quote]
what does he mean by that?

Don’t know if “matte” covers it by 100%: I’m not only trying to put some 2D-image in the background of a scene, but important is the possibility to occlude the 3d-objects partly. I’ll put an example here:

www.augmented.org/temp/combo.jpg

top left: background image, 2D
top right: my 3d-scene with a rotating object
bottom left: a pre-defined 2D-mask
bottom right: my goal I’d like to achieve:

right now I did it “by hand”: capturing the 3d-frame from an offscreen-canvas and checking pixel by pixel if the mask is set to 1 or not. if so, I would occlude my 3d-object… Well, yes. I’d like a stencil buffer. :wink:

So, this is working so far, but the problem of the not-updating-offscreen-canvas remains… I always only see the 1st frame (no renderOffscreenBuffer helps…). If I could do the same now using an onscreen Canvas3D plus a Raster then I’ll be fine, I guess.

Perhaps this helps to point out the problem?

Greetings.

#4

OK, great! The Raster works fine in onscreen-mode.

But can I capture the whole thing as an image (to save it into jpg/avi files)? I’ve read that I have to use an offscreen Canvas3D for that… Or?

… and then I have the update-problem again.

#5

Yup. In film and video this is called a Matte :slight_smile:

If you change the occluded area on a frame by frame basis, its called a “traveling matte”.

:slight_smile:

JK

#6

Hm. Ive never done any off-screen rendering msyelf so I dont’ have an answer on that off the top of my head.

BUT I’ll ask Kevin or Paul to look at your post next time Im in the office and maybe they can either tell you what to try or tel lyo uwhat mre info they need to tel lyou whats wrong…

#7

ok, thanks so far. I’ll play around with the new discovered Raster for now…

I’m puzzled by the constructor of Raster offering a DepthComponent as well. Can I define a z-value for each pixel of my raster, which will affect occlusion? Would be great!! :smiley:
But I’m not sure: the API only mentions the parameter of the DepthComponent with the sentence

[quote]a Raster may be used to readback pixel data (color and/or z-buffer) from the frame buffer in immediate mode.
[/quote]
So this sounds more like only getting the z-buffer but without influencing it…

Best greetings,
Konterfei.

#8

OK. I’m gone for the long weekend (Monday is holiday in spain).

If you can ask about rasters & offscreen I would be deep in your debt. :wink:

Cu.

#9

Yup. I asked Kevin today. hes going to look at your post and either post a response or emai lme a reaspons and ill post it.

Have a great weekend!

JK

#10

[quote]If you can ask about rasters & offscreen I would be deep in your debt. :wink:
[/quote]
Depth rasters can be used for drawing and/or readback. It does give you a separate Z value for each pixel, but you can’t currently use a Raster with both RASTER_DEPTH and RASTER_COLOR at the same time (OpenGL doesn’t support that directly so it was never implemented). This means you can’t really have a separate Z per pixel while you are drawing the raster. As long as you only have one of these mattes, and draw it before everything else in the scene, it shouldn’t matter. I recommend that you try to use two raster objects, one with RASTER_DEPTH and the other with RASTER_COLOR. Draw the depth raster first. Draw the color-only raster second, with depth test and depth write enable disabled. Draw all 3D objects after this and they should be correctly Z-buffered with the image.

For Canvas3D capture, it depends on what you need to do. If all you want is to take a snapshot of the current window at the actual resolution, and if you aren’t worried about other windows occluding the Canvas3D, then you don’t need offscreen rendering. Just read the contents of the window from the postSwap method, and you have your image. If you want to render it at a higher resolution than the current window/screen will allow, or if you don’t want the problem of occluding windows, you will need to use offscreen. Take a look at the PrintCanvas3D example program (you can get it from the https://j3d-examples.dev.java.net project).

– Kevin

#11

Thanks a lot you two. Just reading at home (I know I shouldn’t). :slight_smile:

Will try it tomorrow in the office and will let you know if it works out. Kevin’s information should be sufficient. Sounds promising!

CU.

#12

Hey again.

I tried it out, but so far I get the same result as Mark McKay (just found his posting in the Java3D interest list yesterday) in January, 21th 2004:
http://archives.java.sun.com/cgi-bin/wa?A2=ind0401&L=java3d-interest&F=&S=&P=15470

My code is quite similar. As I understand it I don’t need the RASTER_COLOR Raster for testing purpose (although I tried that as well): I have a Raster using RASTER_DEPTH but the pixels are all flat at the plane attached to the raster’s origin… The moving 3d objects get occluded as soon as they move behind the x-y-plane lying at z-value 0.0f… I expected to see the 3D objects and sometimes (when they move backwards) to see black spots/areas on them – where the raster has depthvalue 0.0f. (0.0f in the depthComponent not being the 3d space position but the closest z-value to the camera, right?).

I used DepthComponentFloat and my depthmask just had values 1.0f and 0.0f… But it does not influence anything.

Without the background RASTER_COLOR the important part looks like this:

BranchGroup objRoot = new BranchGroup();
OrderedGroup orderGrp = new OrderedGroup();


      // this just loads an image into BufferedImage, working fine
      // the image only has black and white areas for defining depthmask
      BufferedImage biMaskTemp = 
            new ImageConverter().readIntoBufferedImage(frame_mask,this);

      // DepthCompononentFloatPlus extends DCFloat by just allowing 
      // the additional definition of the depth data directly from the BufferedImage
      m_depthMask = 
            new DepthComponentFloatPlus(frame_width, frame_height, biMaskTemp);
            
      // holds the image we want to draw, actually we DON'T want to draw it,
      // since this  time we only want to set z-values
      ImageComponent2D img = loadSomeImage();

      // origin of raster lies at z-value 0.0f!
      // m_depthMask is frame_width*frame_height big and has only 1.0fs and 0.0fs inside
      raster = new Raster(new Point3f(-1.0f, 0.75f, 0.0f), 
                      Raster.RASTER_DEPTH, 0, 0, frame_width, frame_height, img, m_depthMask);
      raster.setClipMode(Raster.CLIP_IMAGE);
      

      Appearance app = createAppearance();
      RenderingAttributes ra = new RenderingAttributes();
      ra.setDepthBufferEnable(true);
      ra.setDepthBufferWriteEnable(true);
      app.setRenderingAttributes(ra);

      Shape3D shapeI = new Shape3D(raster, app);
      shapeI.setCapability(Shape3D.ENABLE_PICK_REPORTING);
      shapeI.setBounds(new BoundingSphere(new Point3d(), 0.5));
      transGroupRaster = new TransformGroup();
      transGroupRaster.addChild(shapeI);

orderGrp.addChild(transGroupRaster);

// ... draw 3d geometry and add to orderGrp ...

objRoot.addChild(orderGrp);

Does this code help? Did I get the idea right?
Perhaps you (or someone else) has a hint…

Greetings,
Konterfei.

#13

Hey there.

Any news? I unfortunatelly didn’t get it to work to change the z-value of single raster pixels… they always stick to the origin of the raster… hmmm.

Greetings,
Konterfei.

#14

Sorry been real busy. Ill ask Kevin and/or paul to take another peek at this this week.

JK