Java2D & Native Buffers

Hey!

This is a java2d - opengl related question.

Basically I want to draw into a BufferedImage and pass that image as OpenGL texture.
I know there is a method to create a ByteBuffer from BufferedImage - it looks like this:


byte[] imageArray = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
ByteBuffer imageBuffer = ByteBuffer.wrap(imageArray);

However this won´t work for me, because I need the buffer to be direct, the buffer retrieved
by this method is not direct. Unfortunately my JNI methods (yes thats right - I rolled my own ;D )
won´t accept non direct buffers. A hacky and a performancewise not so optimal solution may
be to copy the non-direct buffer into a direct buffer, but I really wouldn´t like my little program
to spend time on copying :wink:

So is there a way to wrap a BufferedImage around a direct buffer ? Technically that should be
no problem…

Unfortunately, you’ll have to copy the byte[] into a direct buffer yourself.

The problem is, that as long as we cannot pin down objects (the GC must not move these objects in the heap) we can’t make a direct buffer point to these memory-locations. Further, I highly doubt anybody at Sun would be willing to change the public API to give everybody and their dog access to BufferedImage internals, as those seem to get rehauled every few years…

Okay I am a bit frustrated by now !

I found a way to solve that problem.

Sun engineers have nicely abstracted all properties of a graphics surface.

The key class is WritableRaster which uses a DataBuffer. A DataBuffer is an encapsulation of the actual data representation. For example the most used implementation of DataBuffer is the DataBufferByte which is implemented by a byte array. So the way to go is implement an own version of Data Buffer:

[quote]public class DataBufferDirectByte extends DataBuffer {
protected ByteBuffer directByteBuffer;
public DataBufferDirectByte(ByteBuffer directByteBuffer) {
super(TYPE_BYTE, directByteBuffer.capacity());
this.directByteBuffer = directByteBuffer;
}

public int getElem(int bank, int i) {
    return directByteBuffer.get(i);
}

public void setElem(int bank, int i, int val) {
    directByteBuffer.put(i, (byte)val);
}

public ByteBuffer getDirectByteBuffer() {
    return directByteBuffer;
}

}
[/quote]
Very Easy uh ? :wink:
Afterwards one has to wade through lots of other classes, there need to be surfaceformats and colormodels defined before it is finally possible to build an ImageBuffer.

This looks like this:

[quote]int[] offsetarray = {0,1,2,3};
PixelInterleavedSampleModel smodel =
new PixelInterleavedSampleModel(DataBuffer.TYPE_BYTE, 256,256,4,2564,offsetarray);
WritableRaster r = (WritableRaster) Raster.createRaster(smodel,
new DataBufferDirectByte(BufferUtils.allocateDirectByteBuffer(256
4)), new Point(0,0));

        ColorModel m = ImageUtil.createColorModel(r.getSampleModel());
        BufferedImage i = new BufferedImage(m,r,false, null

);
[/quote]
The BufferUtils class was written by me.

But then happens the big #!$5$!*

A class somewhere inside the jdk jungle complains that our newly introduced DataBuffer is not of byte type. What the *!& ? We have defined it to be of byte type by calling the constructor of the superclass with the apropriate parameter;

super(TYPE_BYTE, directByteBuffer.capacity());

The type might be queried then with the DataBuffer.getDataType method.

Then I have a look at the implementation of this “ByteComponentRasters” class and what do i find ???

 
if (!(dataBuffer instanceof  DataBufferByte)) {
                  throw new RasterFormatException(
                           "ByteComponentRasters must have "
                                  + "byte DataBuffers");
              }

That makes me wanna cry ! They actually confused their design ! They are checking the actual class types, instead they should
have been using:
switch(dataBuffer.getDataType);

Oh nooo! I actually think that this is a bug and was not meant this way. No point in abstracting everything, so the system may nicely be extended, and then to allow only one type.

:frowning: >:( >:( >:(

No more motivation to continue…

What do you think about this ?

You could access the BufferedImage array directly from your JNI method?

Hey Tom I am trying to do that ;D

But I believe the only way to access ANYTHING via JNI is to put
it into direct buffers. I would never pass java objects down to c++, for
perfomance reasons. I believe it was a wrong decision from sun
(at least for my purposes) design JNI like it is now. It would have been
better if C++ could directly access java class fields and methods, even
if that would mean recompiling the dll if a new java version comes
around.

If I remember correctly the issue with accessing java arrays from c++
is that the arrays gets copied or something. I will have a look at that later.
Have to go to uni,

Thanks
Frederick

[quote=""]
If I remember correctly, instanceof checks the whole inheritance tree. So you could just write “extends DataBufferByte”. It’s not clean, but hey it should work :wink:

There is a way to access java arrays from JNI c code without the array being copied. There is a special array mehtod you can use from JNI. I think it’s called getPrimitiveArrayCritical, or something like that. The VM has to support pinning and a few other restrictions. But you should atleast check it out.

EDIT: http://download.java.net/jdk7/docs/technotes/guides/jni/spec/functions.html (search for GetPrimitiveArrayCritical)

Hey!

I wouldn´t really like that, because I switched to Java because I want to program nice and cleanly :wink:
That way I would inherit an array data structure and methods I don´t need - kind of a mess.

[quote]There is a way to access java arrays from JNI c code without the array being copied. There is a special array mehtod you can use from JNI. I think it’s called getPrimitiveArrayCritical, or something like that. The VM has to support pinning and a few other restrictions. But you should atleast check it out.
[/quote]
Thanks for the link, I might give that a try. I plan to use only modern VMs so no problem with pinning I hope. Still I would like to use the buffers, but…

Maybe I should file this as a bug report ? I would tend to call it a bug, because it is definetly not a feature lol Also it would please me to know, that I can render to any surface I like and the jdk supports this functionality - unfortunately only theoretically.

It is not a bug. You need to define your own Raster type to handle your DataBuffer.

However, the problem is that even when you make it all work - that is,
create a BufferedImage with your direct data buffer, the only way
Java2D would be able to render to/from it is using get/setElem,
which will be excruciatingly slow - since Java2D doesn’t know
how to access your data other than through get/set elem methods.

What we really need is proper support for DataBuffers with out of heap
data placement. I believe there’s an RFE somewhere already.
Unfortunately this will have to wait at least until Java 7.

Dmitri

[quote]It is not a bug. You need to define your own Raster type to handle your DataBuffer.
[/quote]
Okay that´s too much work then. I had thought, because the raster has a pluggable DataBuffer it would suffice to exchange it.

[quote]Java2D would be able to render to/from it is using get/setElem,
which will be excruciatingly slow
[/quote]
I really hope the JVM will inline getter/setter methods ?

[quote]What we really need is proper support for DataBuffers with out of heap
data placement.
[/quote]
I would really appreciate stack allocation too, but i don´t understand how it could help in this case. ??? ??? :slight_smile:

…I think I will stick with toms suggestion - everything else seems to be too complicated…

Implemented the method with “getPrimitiveArrayCritical” - works fine so far, but of course I can´t check if an internal copy is made - lets hope the best :wink:

Why don’t you just do this?


//image needs to be initialized as new BufferedImage(width, height, BufferedImage.TYPE_4BYTE_ABGR);
byte[] imageArray = (byte[]) image.getRaster().getDataElements(0, 0, image.getWidth(), image.getHeight(), null);
ByteBuffer imageBuffer = ByteBuffer.allocateDirect(imageArray.length);
imageBuffer.put(imageArray);

As long as you use the correct type in the constructor you can cast it to a byte array.

Because we wanted to avoid that data-copy…

It’s not only the cost of getter/setter methods, it’s the fact that with such
image we’ll use the most generic rendering loops available, written
in java. The vm isn’t yet up to the task of competing with highly optimized
native code (which is used when one of the image types we know
about is used) in this case.

For example, a fillRect will translate to tons of setElem calls (one per pixel),
while for ‘normal’ image it will be a bunch (or a single one) of quick memsets.

Dmitri

Ahh okay I see ;D

I had always thought that rasterization algorithms were written in java, because people
referred to swing as a pure java solution and being therefore more sluggish than SWT, which
has native drawing code.
So that is not entirely true.

Good to hear that Java2D uses native code :slight_smile: I also believe that native code is superior for that
kind of things, because one can use SIMD, which is cool, one can care for memory alignment and
cache utilization and of course graphics hardware acceleration.

So thank you all - you are awesome
wish me luck, so that I actually get something done :wink:
Frederick

[EDIT: major orthographic corrections ::)]

Just a note: ideally we would have all our loops written in Java and
let hotspot worry about proper translation to machine code
(using SIMD or whatever is available) - at least, when rendering
to a heap-based image (like BufferedImage).

We do have a project
which works in this direction - but since it requires a cooperation from
the super-busy vm team, it’s not progressing much…

Dmitri

Hi trembovetski :slight_smile:

I heard that auto-vectorization is really a hard problem and that actually no existing compiler is good at it at the moment. Good to hear that there is work done on it. Personally I wouldn´t mind to write that code by myself, maybe for a particle simulation, skinning or something like that. Maybe java performance is just good enough - time will show - I haven´t pushed my machine to the limit up to now ;D
But when time has come it would be cool to have the opportunity.
Then there is always the tradeoff between speedup and native calling overhead, caused by JNI, to be considered. I suppose SIMD matrix multiplication wouldn´t be much faster than a java version because of that !?

What I really would appreciate is some improved capability to talk to native code, it would be COOL when java could at least partially have memory layout compatible to c++, maybe using structs or something like that. I had to do some workarounds in that area. But I understand that this is not the main application area of java.

Thank you for your nice comments !

There was some discussion about pinning in the VM: As far as I can say nobody should expect pinning to be available anymore.
Hotspot itself does not support pinning, as it would destroy the bump-the-pointer allocation strategy - and because allocation for typical java programs is much more important than java->C data transport I don’t think things will change.

The get*Critical Methods themself block the GC, therefor allow to access the data without copying it.

By the way, my old Athlon-Thunderbird-1Ghz (PC100 ram) copied arround with about 800mb/s - so nothing to really worry about anymore :slight_smile:

lg Clemens

I have been told that using Get/SetArrayRegion may actually be better than
GetPrimitive* .

You might want to try it.

Hey Trembovetski - Thanks :wink:

I will definately give that a try… at the moment I am quite busy with learning, so that will have to wait a little. Big thanks, that you asked somebody for it :smiley:

@Linuxhippy: 800mb/s seems impressive first but it is not so if you divide that value by the fps-count. Lets say by 40 - then we will have 20MB per frame. For 2D Animation like video etc. one has send data constantly to the gpu (I am not sure that I want to do THAT ;)). Things like this will keep adding up if one does not care. I am not a hardcore optimizer and i would always prefer simplicity rather than speed - but needless copying is something one really should avoid, I think.