Can I scan the screen or get a screenshot of my screen within Java?

Newbie & Debugging Questions
#1

I’m making a program to save certain keyboard or mouse actions you have done on your computer and reproduce them for you, helping out with a lot of things like mundane multimedia tasks and whatnot. I want you to be able to provide conditionals that will cause the program to produce certain actions, and these conditionals I want to rely solely on what is being displayed on your monitor.

Like, for example, I want the program to automatically log in to something for me once it opens, like JGO. It would need to scan the screen to see if it perhaps had the JGO banner in the expected place. If it did, it would automatically click in the login box, type out my name, then the password box and type out my password.

Yes, pretty ambitious, but I’ve got pretty much everything working already. I’m just wondering if there’s some way to return what’s being displayed as a BufferedImage, or to get a screenshot of it, at least, which may be a little processor intensive but it would certainly work.

Any ideas? Thanks guys.

#2

Robot.createScreenCapture(Rectangle)

#3

Amazingly, I was already using Robot and didn’t see that. Thanks very much.

Any idea how costly this operation is?

#4

try out this test:


import java.awt.*;
public class ScreenTest {
	public static void main(String args[]) {
		try {
			long now = System.currentTimeMillis();
			long ms = 0;
			Robot r = new Robot();
			Dimension d = Toolkit.getDefaultToolkit().getScreenSize();
			Rectangle rect = new Rectangle(0,0,d.width,d.height);
			int count = 0;
			while (ms < 5000) {
				r.createScreenCapture(rect);
				ms+=(System.currentTimeMillis()-now);
				now = System.currentTimeMillis();
				count++;
			}
			System.out.println(count+" screenshots taken in "+(ms/1000.0)+" seconds");
		}
		catch (AWTException e) {
			e.printStackTrace();
		}
	}
}

for me I get about 14 screen captures per second on a 1024x768 screen. if you already have a presumed position for, say, the JGO banner, you can just use a smaller Rectangle for the screen capture and it should be alot faster than taking an ENTIRE screenshot

#5

Ooh yeah, that’s working really fast. I just have the user specify a certain rectangle that they want to search for, generally not larger than 50x50 pixels. 50x50 pixels goes really really fast. I can also put in the option to scan the screen for certain patterns, and therefore it would work across multiple resolutions, at least somewhat (look for the text box pattern, because it will always be the same size, etc.)

I wonder if there would be any way to get this to work flawlessly aside from forcing a resolution change. The windows will obviously move around and be in different places depending on resolution, even depending on computer. I suppose I could do something called findWindow() function that finds the window in question so it is only performed once to get a relative position, then smaller screen analysis can be performed. But can you think of another way to do this?

#6

Hook into the win32 api?!

Grabbing a screenshot and trying to derive what is where is a completely backward way of doing it.

You code is going to be platform specific anyway, so you might as well make use of the underlying api’s provided to fulfill the exact feature you desire.

You never know, there may already be a java wrapper around for querying the active processes and/or window handles that are currently active.

#7

abuse is right, there are 100’s of things that can get in the way with the screen capture method. the user can change the size of the titlebar text, which can throw everything off, for example…

#8

Yeah, this is wholly true, but even if I could contact Windows does that mean that I could have Internet Explorer return to me its window size, font size, locations of text fields, etc.? I feel like even if I could do that it would just because program specific. I want this to be functionable through any program as long as the user saves a set of actions depending on different highlights on their screen.