Software rendering

Hello, I’m new here.
I’ve put together a game which uses software rendering, something like this:


// Create image
Component target = ...; // A JPanel inside a JFrame's contentPane()
BufferedImage image = (BufferedImage) target.createImage(width, height);
int[] buffer = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();

// Software rendering
for (...)
{
    buffer[i] = ...;
}

// Blit it to screen
target.getGraphics().drawImage(image, 0, 0, target.getWidth(), target.getHeight(), null);

This works ok, but I noticed a significant frame rate drop when the window gets bigger.

Back in the days when I used DirectDraw, I usually just did something like this this:

  1. ddsSecondary->Lock()
  2. Copied the pixels
  3. ddsSecondary->Unlock()
  4. ddsPrimary->Blt(…)
    and that always worked fine, regardless of the window size.

I assume the difference is that in my Java code, the surface remains locked. What can I do to improve performance?

Thanks,
Martin

Go read up on BufferStrategy. It handles the page flipping automatically. i.e. You don’t need to lock and unlock surfaces. :slight_smile:

Thanks for your help, I did some reading and added BufferStrategies to my app. But it appears the problem is related to the scaling inside drawImage(…). Here’s a test app: (assumes your desktop is 32 bpp)


import java.awt.Canvas;
import java.awt.Graphics;
import java.awt.Rectangle;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;

import javax.swing.JFrame;

public class Test
{
	public static void main(String[] args)
		throws Exception
	{
		/* Window */
		JFrame window = new JFrame();
		window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		Rectangle bounds = new Rectangle(10, 10, 0x300, 0x300);
		window.setBounds(bounds);
		window.setVisible(true);

		/* Canvas, strategy */
		Canvas canvas = new Canvas();
		window.getContentPane().add(canvas);
		canvas.createBufferStrategy(3);
		BufferStrategy bufferStrategy = canvas.getBufferStrategy();

		/* Image, buffer */
		BufferedImage image = (BufferedImage) canvas.createImage(window.getWidth(), window.getHeight());
		int[] buffer = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();

		int color = 0;
		while (true)
		{
			/* Pixels */
			for (int i = 0; i < buffer.length; i++, color++)
			{
				buffer[i] = color | 0xff000000;
			}

			/* Blit to screen */
			Graphics g = bufferStrategy.getDrawGraphics();
			g.drawImage(image, 0, 0, window.getWidth(), window.getHeight(), null);
			bufferStrategy.show();
			g.dispose();
		}
	}
}

Note how everything slows down as the window gets bigger. I guess this is because the dd surface for that BufferedImage must remain locked at all times since Java allows me to access its pixels directly at any time.

Is there a better way to use a BufferStrategy and still get direct access to the pixels?
Or is there a way to “unlock” the BufferedImage so that drawImage(…) can be hardware accelerated?
Or did I just misunderstand everything and now I’m looking like an idiot? :slight_smile:

/Martin

You don’t manipulate pixels in Java. Doing so is going to get you all the performance of glDrawPixles. i.e. Absolutely horrible performance.

What you want is to blit your images directly (as if they were polygon surfaces), then apply transformations for scaling, rotation, and special effects. This will then get shunted down the Direct3D/OpenGL pipeline to be rendered directly onto the backbuffer in Video RAM.

For maximum performance, make sure you acquire an image that matches the depth and memory organization of the backbuffer before rendering. Otherwise Java will convert the image to the native format every time you need to render the image. As you can imagine, this is EXTREMELY slow. As a result, you’ll usually want to create a loader that automatically copies the image data into new images of the proper format. Such a loader is available here:

http://wiki.java.net/bin/view/Games/LoadingSpritesWithImageIO

You may also find that perusing this tutorial may help answer some of your questions:

http://www.cokeandcode.com/info/tut2d.html

And stop worrying about surface locking. It’s got nothing to do with anything. :slight_smile:

That’s the whole point - I do need to manipulate pixels directly, hence the topic “Software rendering”. I know this is not normal but in this case I really need to do it. I’ve got all the pixels ready and I just need to blit them to the primary surface as fast as possible. :slight_smile:

WIth DirectDraw, I just did the regular [lock, copy pixels, unlock] thing and that way I could use hardware accelerated blits for that surface but still access its pixels directly for a limited time (during the lock). Is there absolutely no way to do this with Java?
I assume calling “((DataBufferInt) image.getRaster().getDataBuffer()).getData();” results in hardware acccelerated blits being disabled since from that point on I can access the pixels directly at any time, so the surface is most likely moved out of VRAM. What I want is to tell Java “Ok, I don’t need direct access to the pixels anymore” so that I can have hardware acceleration enabled again for that surface. How can I do that?

/Martin

You might want to ask this over in the 2D section. Normally we discourage cross posting but I dont know if the Java2D guys read this and i know they DO read the 2D topic.

It’s inefficient to access the Video RAM pixel by pixel. The result would be tons of unnecessary bandwidth on the line. Thus Java copies the entire buffer when it’s done. What you’re doing is pretty close to the fastest thing available . A few helpful hints:

  • Create a Graphics Configuration compatible image for your backbuffer the same way I pointed to above.(i.e. BufferedImage buffer = gc.createCompatibleImage(width, height); ) This will prevent Java from having to convert the image data.
  • Set the BufferCapabilities to BufferCapabilities.FlipContents.PRIOR to save time during flips.

That’s about as fast as you’re going to get. You should be able to get 60 or so FPS, though.

Keep in mind that this is all going through a 3D API, so pixel level performance is not intended to be fast.

Perhaps if you could tell us more about why you specifically need pixel level rendering, we could offer more suggestions? Quite a few new Java Game programmers don’t realize the things that can be done without access to the pixel level.

I am guessing for software 3d.

use:

int[] imgData = (int[]) ((DataBufferInt) img.getRaster().getDataBuffer()).getData();

to get the array of pixels that make up a buffered image, (the type of databuffer / array will vary depending on which type of image it is, the above works the BufferedImage.TYPE_INT_RGB at least.

Performance is not too bad using this method, i am able to get ~300fps with a 512*384 img just clearing it and drawing it to the screen every frame.

I appreciate your help, guys.

I’m writing a console emulator (NES/Famicom). All pixels are generated by the GPU emulator, I’m afraid there’s no way to use hardware acceleration in this case. On each frame, the GPU emits a 256x240 image. I need to use hardware acceleration when blitting that image to screen.

[quote]Performance is not too bad using this method, i am able to get ~300fps with a 512*384 img just clearing it and drawing it to the screen every frame.
[/quote]
It works, but for a big window this method is several times slower than using hardware accleration for that blit. That’s why I’m looking for alternatives… :-\

/Martin

[quote]I’m writing a console emulator (NES/Famicom). All pixels are generated by the GPU emulator, I’m afraid there’s no way to use hardware acceleration in this case.
[/quote]
That’s not actually true. The NES/Famicom contained sprite hardware. By rendering the sprite slivers to correctly sized buffered images, you could potentially accelerate the process. Especially if you cached sprite/tile values. It’s a lot more complicated method than a straightup NTSC GPU emulator, though. :frowning:

[quote] On each frame, the GPU emits a 256x240 image. I need to use hardware acceleration when blitting that image to screen. It works, but for a big window this method is several times slower than using hardware accleration for that blit.
[/quote]
Are you resizing the image before or during the render? Because I guarantee that the fastest method is to render to a 256x256 buffered image (just ignore the extra area) then scale it with the g.drawImage(image, x, y, width, height, null) method. That will transfer less data over the bus, and force the hardware to do the hard work for you. :slight_smile:

That’s not actually true. The NES/Famicom contained sprite hardware. By rendering the sprite slivers to correctly sized buffered images, you could potentially accelerate the process. Especially if you cached sprite/tile values. It’s a lot more complicated method than a straightup NTSC GPU emulator, though. :frowning:
[/quote]
Yeah, I’ve thought about that too. I know it all may seem simple until you look closer into the tech details. But believe me, hardware rendering is not a good (or even fast) solution here. I’m 100% sure on this one.

Are you resizing the image before or during the render? Because I guarantee that the fastest method is to render to a 256x256 buffered image (just ignore the extra area) then scale it with the g.drawImage(image, x, y, width, height, null) method. That will transfer less data over the bus, and force the hardware to do the hard work for you. :slight_smile:
[/quote]
Yeah, that’s exactly what I want to do - I want drawImage to be hw accelerated after I’ve set the pixels manually. Do you have any working example code? Could you modify this example so that it forces the hardware to do the hard work for me?


import java.awt.Canvas;
import java.awt.Graphics;
import java.awt.Rectangle;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import javax.swing.JFrame;

public class Test
{
	public static void main(String[] args)
		throws Exception
	{
		/* Window */
		JFrame window = new JFrame();
		window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		Rectangle bounds = new Rectangle(10, 10, 0x200, 0x200);
		window.setBounds(bounds);
		window.setVisible(true);

		/* Canvas, strategy */
		Canvas canvas = new Canvas();
		window.getContentPane().add(canvas);
		canvas.createBufferStrategy(2);
		BufferStrategy bufferStrategy = canvas.getBufferStrategy();

		/* Image, buffer */
		BufferedImage image = (BufferedImage) canvas.createImage(0x100, 0x100);
		int[] buffer = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();
		int color = 0;

		while (true)
		{
			/* Pixels */
			for (int i = 0; i < buffer.length; i++, color++)
			{
				buffer[i] = color | 0xff000000;
			}

			/* Blit to screen */
			Graphics g = bufferStrategy.getDrawGraphics();
			g.drawImage(image, 0, 0, window.getWidth(), window.getHeight(), null);
			g.dispose();
			bufferStrategy.show();
		}
	}
}

Doh! I didn’t think to look at the code that you posted to see if it was already doing the resize. It looks fine by me. Here’s a version with a frame counter added in:

import java.awt.Canvas;
import java.awt.Color;
import java.awt.Graphics;
import java.awt.Rectangle;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import javax.swing.JFrame;

public class Test
{
    public static void main(String[] args)
    throws Exception
    {
        /* Window */
        JFrame window = new JFrame();
        window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        Rectangle bounds = new Rectangle(10, 10, 0x200, 0x200);
        window.setBounds(bounds);
        window.setVisible(true);

        /* Canvas, strategy */
        Canvas canvas = new Canvas();
        window.getContentPane().add(canvas);
        window.invalidate();
        window.validate();
        canvas.createBufferStrategy(2);
        BufferStrategy bufferStrategy = canvas.getBufferStrategy();

        /* Image, buffer */
        BufferedImage image = (BufferedImage) canvas.createImage(0x100, 0x100);
        int[] buffer = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();
        int color = 0;

        long time = 0;
        long delta = 0;
        int counter = 0;
        long fps = 0;

        while (true)
        {
            /* Pixels */
            for (int i = 0; i < buffer.length; i++, color++)
            {
                buffer[i] = color | 0xff000000;
            }

            /* Blit to screen */
            Graphics g = bufferStrategy.getDrawGraphics();
            g.drawImage(image, 0, 0, window.getWidth(), window.getHeight(), null);
            g.setColor(Color.white);
            g.drawString(String.valueOf((int)fps), 10, 10);
            g.dispose();
            bufferStrategy.show();

            delta += System.currentTimeMillis() - time;
            time = System.currentTimeMillis();
            counter++;

            if(counter >= 100)
            {
                fps = 100000/delta;

                delta = 0;
                counter = 0;
            }
        }
    }
}

I get about 400 frames per second on my machine. I actually added “try {Thread.sleep( 8 );} catch(Exception e) {}” right after “bufferStrategy.show();” to lock it at 100FPS so I could appreciate the effect. (Ahh, lovely. :)) Unless you’re on a Mac, I wouldn’t be surprised if you’re seeing similar results.

Yeah, I get about 400fps too, but it’s still not hw accelerated - it’s using my precious cpu time to scale that image.
If I resize the window to its smallest size, I get 1700 fps.
If I maximize the window, the frame rate drops to 120fps.
With hw acceleration, I believe we would see >1700fps no matter what the window size.

In my real app - which is somewhat more complex :slight_smile: - these numbers are 250fps/90fps. This suggests that if I could have drawImage(…) hw accelerated, I could triple my app’s performance.

  1. set one of the system properties - sun.java2d.opengl=true, or sun.java2d.ddscale=true.
  2. perform all your per pixel operations to a BufferedImage compatible with the screen pixel format (obtained from createCompatibleImage)
  3. copy the image to a VolatileImage.
  4. blit the VolatileImage with a scaling transform, onto your BufferStrategy.

This should result in the scale being performed in hardware, not software.

Though tbh, this shouldn’t be necessary - the scale should be getting done in hardware anyway.
If scaling an unaccelerated BufferedImage, onto an accelerated surface means the scale is done in software, not hardware - I would consider it a bug in the java2d implementation.

Depressingly enough, I’m right - you have to copy it to a VolatileImage, before performing the scale - otherwise the scale won’t be hardware accelerated.

Also, rather depressingly - it appears drawImage(img, x,y,width,height,imgObs) isn’t accelerated by ddscale.
You have to use an AffineTransform.

Heres the code, minor adapation from above.

I get a solid 1050fps when windowed, 950 when the window is maximised.


import java.awt.Canvas;
import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.GraphicsConfiguration;
import java.awt.Rectangle;
import java.awt.geom.AffineTransform;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import java.awt.image.VolatileImage;

import javax.swing.JFrame;

public class Test
{
    public static void main(String[] args)
    throws Exception
    {
    	System.setProperty("sun.java2d.ddscale","true");
    	//System.setProperty("sun.java2d.opengl","true");
//    	 opengl pipeline doesn't work properly for me at all,
//       renders vsync'ed in a window, doesn't resize the BufferStrategy when the canvas is rescaled,
//       and causes a VM crash when I close the application down :S
    	
        /* Window */
        JFrame window = new JFrame();
        window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        Rectangle bounds = new Rectangle(10, 10, 0x200, 0x200);
        window.setBounds(bounds);
        window.setVisible(true);

        /* Canvas, strategy */
        Canvas canvas = new Canvas();
        window.getContentPane().add(canvas);
        window.invalidate();
        window.validate();
        canvas.createBufferStrategy(2);
        BufferStrategy bufferStrategy = canvas.getBufferStrategy();

        GraphicsConfiguration gc = canvas.getGraphicsConfiguration();
        
        /* Image, buffer */
        VolatileImage vi = gc.createCompatibleVolatileImage(0x100,0x100);
        
        BufferedImage image = gc.createCompatibleImage(0x100, 0x100);
        int[] buffer = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();
        int color = 0;

        long time = 0;
        long delta = 0;
        int counter = 0;
        long fps = 0;

        while (true)
        {
            /* Pixels */
            for (int i = 0; i < buffer.length; i++, color++)
            {
                buffer[i] = color | 0xff000000;
            }

            /* Blit to screen */
            Graphics2D g = (Graphics2D)bufferStrategy.getDrawGraphics();
            vi.getGraphics().drawImage(image,0,0,null);
            
            AffineTransform at = g.getTransform();
            g.setTransform(AffineTransform.getScaleInstance(canvas.getWidth()/(double)vi.getWidth(), canvas.getHeight()/(double)vi.getHeight()));
            
            g.drawImage(vi, 0, 0, null);
            
            g.setTransform(at);
            g.setColor(Color.white);
            g.drawString(String.valueOf((int)fps), 10, 10);
            g.dispose();
            bufferStrategy.show();

            delta += System.currentTimeMillis() - time;
            time = System.currentTimeMillis();
            counter++;

            if(counter >= 100)
            {
                fps = 100000/delta;

                delta = 0;
                counter = 0;
            }
        }
    }
}

Anon666, your code did the trick and everything works great now - I went from 90fps to 240fps in my app!!. :slight_smile:
Below is some more test code, you can try different stretch strategies by clicking the canvas.

I got the same fps without AffineTransform as I got with it. I have no idea why you didn’t, it seems reasonable for drawImage to be hw accelerated for VolatileImages. :-\

Thanks everyone for your help.


import java.awt.Canvas;
import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.GraphicsConfiguration;
import java.awt.Rectangle;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
import java.awt.geom.AffineTransform;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import java.awt.image.VolatileImage;
import javax.swing.JFrame;

public class Test
{
	private static int stretchStrategy;

	public static void main(String[] args)
		throws Exception
	{
		System.setProperty("sun.java2d.ddscale", "true");

		/* Window */
		JFrame window = new JFrame();
		window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		Rectangle bounds = new Rectangle(10, 10, 0x200, 0x200);
		window.setBounds(bounds);
		window.setVisible(true);

		/* Canvas, strategy */
		Canvas canvas = new Canvas();
		window.getContentPane().add(canvas);
		window.invalidate();
		window.validate();
		canvas.createBufferStrategy(2);
		BufferStrategy bufferStrategy = canvas.getBufferStrategy();
		GraphicsConfiguration gc = canvas.getGraphicsConfiguration();
		canvas.addMouseListener(new MouseListener()
		{
			public void mouseClicked(MouseEvent e)
			{
				stretchStrategy++;
				stretchStrategy %= 3;
			}

			public void mousePressed(MouseEvent e)
			{

			}

			public void mouseReleased(MouseEvent e)
			{

			}

			public void mouseEntered(MouseEvent e)
			{

			}

			public void mouseExited(MouseEvent e)
			{

			}
		});

		/* Image, buffer */
		VolatileImage vi = gc.createCompatibleVolatileImage(0x100, 0x100);
		BufferedImage bi = gc.createCompatibleImage(0x100, 0x100);
		int[] buffer = ((DataBufferInt) bi.getRaster().getDataBuffer()).getData();

		int color = 0;
		long time = 0;
		long delta = 0;
		int counter = 0;
		long fps = 0;

		while (true)
		{
			/* Pixels */
			for (int i = 0; i < buffer.length; i++, color++)
			{
				buffer[i] = color | 0xff000000;
			}

			/* Blit to screen */
			Graphics2D g = (Graphics2D) bufferStrategy.getDrawGraphics();

			switch (stretchStrategy)
			{
			case 0:
			{
				/*
				 * Copy the BufferedImage to a VolatileImage, then stretch the
				 * VolatileImage to screen with an AffineTransform
				 */
				vi.getGraphics().drawImage(bi, 0, 0, null);
				double width = canvas.getWidth() / (double) vi.getWidth();
				double height = canvas.getHeight() / (double) vi.getHeight();
				AffineTransform at = g.getTransform();
				g.setTransform(AffineTransform.getScaleInstance(width, height));
				g.drawImage(vi, 0, 0, null);
				g.setTransform(at);
				break;
			}
			case 1:
			{
				/*
				 * Copy the BufferedImage over to a VolatileImage and then stretch the
				 * VolatileImage to screen.
				 */
				vi.getGraphics().drawImage(bi, 0, 0, null);
				g.drawImage(vi, 0, 0, canvas.getWidth(), canvas.getHeight(), null);
				break;
			}
			case 2:
			{
				/*
				 * Stretch the BufferedImage directly to screen
				 */
				g.drawImage(bi, 0, 0, canvas.getWidth(), canvas.getHeight(), null);
				break;
			}
			}

			g.dispose();
			bufferStrategy.show();

			/* FPS counter */
			delta += System.currentTimeMillis() - time;
			time = System.currentTimeMillis();
			counter++;
			if (counter >= 100)
			{
				fps = 100000 / delta;
				String title = "Stretch strategy: " + stretchStrategy + ", FPS: " + String.valueOf((int) fps);
				window.setTitle(title);
				delta = 0;
				counter = 0;
			}
		}
	}
}

Since you’re all big boys and girls and understand that flags aren’t guaranteed to be supported forever,
I’ll post another version with some flags which have been there for a while.
If you’re curious what those flags mean, feel free to browse the jdk source code =)


import java.awt.Canvas;
import java.awt.Color;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.Rectangle;
import java.awt.geom.AffineTransform;
import java.awt.image.BufferStrategy;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
import java.awt.image.VolatileImage;

import javax.swing.JFrame;

public class Test
{
    public static void main(String[] args)
    throws Exception
    {
        boolean useVI = false;
        for (String s : args) {
            if (s.equals("-usevi")) {
                useVI = true;
            } else if (s.equals("-usebi")) {
                useVI = false;
            }
        }
        System.setProperty("sun.java2d.ddscale", "true");
        if (!useVI) {
            // this should work with opengl pipeline as well
//            System.setProperty("sun.java2d.opengl", "true");
            
            // d3d is only accelerated in mustang
//            System.setProperty("sun.java2d.d3d", "true");

            // magic passes
            System.setProperty("sun.java2d.accthreshold", "0");
            System.setProperty("sun.java2d.allowrastersteal", "true");
        }
        /* Window */
        JFrame window = new JFrame();
        window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        Rectangle bounds = new Rectangle(10, 10, 0x200, 0x200);
        window.setBounds(bounds);
        window.setVisible(true);

        /* Canvas, strategy */
        Canvas canvas = new Canvas();
        window.getContentPane().add(canvas);
        window.invalidate();
        window.validate();
        canvas.createBufferStrategy(2);
        BufferStrategy bufferStrategy = canvas.getBufferStrategy();

        /* Image, buffer */
        BufferedImage image = (BufferedImage) new BufferedImage(0x100, 0x100, BufferedImage.TYPE_INT_RGB);
        int[] buffer = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();
        int color = 0;
        Color translucentColor = new Color(0.0f, 0.0f, 0.0f, 0.0f);
        
        long time = 0;
        long delta = 0;
        int counter = 0;
        long fps = 0;

        VolatileImage vi = 
            useVI ? canvas.createVolatileImage(0x100, 0x100) : null;
        
        while (true)
        {
            /* Pixels */
            for (int i = 0; i < buffer.length; i++, color++)
            {
                buffer[i] = color | 0xff000000;
            }

            /* Blit to screen */
            Graphics2D g = (Graphics2D)bufferStrategy.getDrawGraphics();
            if (useVI) {
                vi.validate(canvas.getGraphicsConfiguration());
                vi.getGraphics().drawImage(image, 0, 0, null);
                g.drawImage(vi, 0, 0, 
                            window.getWidth(), window.getHeight(), null);
                
            } else {
                // mark the image dirty so the accelerated copy is updated
                Graphics gg = image.getGraphics();
                gg.setColor(translucentColor);
                gg.fillRect(0, 0, 1, 1);
                
                AffineTransform at = g.getTransform();
                g.setTransform(
                    AffineTransform.getScaleInstance(
                        canvas.getWidth()/(double)image.getWidth(), 
                        canvas.getHeight()/(double)image.getHeight()));
           
                g.drawImage(image, 0, 0, null);
           
                g.setTransform(at);
            }
            g.setColor(Color.white);
            g.drawString(String.valueOf((int)fps), 10, 10);
            g.dispose();
            bufferStrategy.show();

            delta += System.currentTimeMillis() - time;
            time = System.currentTimeMillis();
            counter++;

            if(counter >= 100)
            {
                fps = 100000/delta;

                delta = 0;
                counter = 0;
            }
        }
    }
}

yeah, I am now too =)

I guess I was just mistaken :wink: