How to differenciate mouse events from the user and those from java.awt.Robot?

gouessej · November 9, 2009, 4:36pm

Hi!

I’m looking for a way of making the difference between user mouse events and mouse events generated by java.awt.Robot. In AWTEvent, there is an attribute “isSystemGenerated” but it is always at true in my case Robot uses a peer (WRobotPeer, MRobotPeer, XRobotPeer). I would like to find a way of modifying the events that the robot creates in order to handle them differently in several engines that need a mouse look and that use AWT. Do you have an idea? Best regards. Thank you for your attention.

Riven · November 9, 2009, 11:28pm

AFAIK, it can’t be done.

Robot hooks into the OS, with the sole purpose of being indistinguishable from real user input.

Therefore you cannot modify the AWTEvents created by the Robot, because the Robot does not create AWTEvents. The Robot is operating at a lower level, and eventually the JVM receives a signal from the OS and creates the AWTEvent. The JRE has no way of figuring out what caused the event.

You can however build heuristics and make assumptions on what is likely and what is not. If you told Robot to move the mouse to X,Y and you receive an event of the mouse being at X,Y you can be fairly sure you caused that event. If the mouse is not exactly at X,Y you can be fairly sure the user moved the mouse. This way you can simulate a grabbed mouse.

But… what are you trying to do? And are you sure Robot is the best way to achieve it?

gouessej · November 10, 2009, 12:35am

[quote=“Riven,post:2,topic:34508”]
Some years ago, bienator and you helped me in fixing this bug in TUER. I want to handle the continuous mouse look in Ardor3D without JInput, only with AWT. Currently, when the mouse reaches an edge of the screen, you cannot turn anymore. Your heuristic solution may be a fine solution, thank you very much.

Do you think I could write a RFE in order to ask Sun/Oracle for help? I think it wouldn’t be efficient.

DzzD · November 10, 2009, 12:40am

ho! I remember I did something like for a FPS demo (http://demo.dzzd.net/FPSSample9/) I will post it if I can found source code

gouessej · November 10, 2009, 12:44am

I have done it too in TUER but with my own engine, it was less difficult to implement than in other real 3D engines. Maybe you found another more interesting solution

DzzD · November 10, 2009, 12:45am

got it ! why do you want to differentiate Robot than other ??

also, I commented the extension loader part as it wont be needed for you (it is only requiered to load from an unsigned applet)

first create an instance (new FPSMouseControler )
second set the wanted canvas to control (setCanvas)

third read dmx & dmy to get mouse delta X and mouse delta Y

	public class FPSMouseControler implements MouseMotionListener 
	{
		int mx=0;
		int my=0;
		int screenX;
		int screenY;
		int screenXDiv2;
		int screenYDiv2;
		public int dmx=0;
		public int dmy=0;
		Canvas c;
		
		Robot robot;
		//IExtensionLoader loader;

		public FPSMouseControler() throws Exception
		{
		
				//if(loader == null)
				//	loader = ExtensionLoader.getLoader(this.getClass());

				robot = new Robot();
				Toolkit t = Toolkit.getDefaultToolkit();
				screenX = t.getScreenSize().width;
				screenY = t.getScreenSize().height;
				screenXDiv2=screenX>>1;
				screenYDiv2=screenY>>1;
				
				
				

				
		}
		
		public void setCanvas(Canvas c)
		{
			if(this.c!=null)
				this.removeCanvas(this.c);
				
			this.c=c;
			c.addMouseMotionListener(this);	
			
				//try to hide cursor using an internet grabbed source code... lazy me :)
				try
				{
				
				int[] pixels = new int[16 * 16];
				Image image = Toolkit.getDefaultToolkit().createImage(
				new MemoryImageSource(16, 16, pixels, 0, 16));
				Cursor transparentCursor =
				Toolkit.getDefaultToolkit().createCustomCursor
				     (image, new Point(0, 0), "invisibleCursor");
				 this.c.setCursor(transparentCursor);
             }
             catch(Throwable te)
             {
             	te.printStackTrace();
             	//dont car if we cant hide it
             }			
		}
		
		public void removeCanvas(Canvas c)
		{
			c.removeMouseMotionListener(this);	
			this.c=null;
			c.setCursor(Cursor.getPredefinedCursor(Cursor.CROSSHAIR_CURSOR));
			
		}
		
		
		public void mouseDragged(MouseEvent e) 
		{
			this.mouseMoved(e);
		}
        
		public void mouseMoved(MouseEvent e)  
		{
			mx=e.getXOnScreen();
			my=e.getYOnScreen();
			if(mx!=screenXDiv2 || my!=screenYDiv2)
			{
				dmx+=mx-screenXDiv2;
				dmy+=my-screenYDiv2;	
				
				robot.mouseMove(screenXDiv2, screenYDiv2);
			}
		}


	}

EDIT : pom pom pom

DzzD · November 10, 2009, 12:46am

try the demo, It work pretty well (hit “M”)

gouessej · November 10, 2009, 9:31am

It works pretty well in your demo but the source code looks like my first implementation of mouse look; in some cases, it doesn’t work, I don’t know why Bienator & Riven’s solution has worked better than this for me.

Anyway, if I don’t want to rewrite the whole ardor3d-awt module, the most realistic solution is the latest Riven’s suggestion, the heuristic one.

DzzD · November 10, 2009, 9:38am

I cant see a case where it wont works ? any sample ? (maybe the aonly problem could be that you have to adjust screenWDiv2 and screenHeightDiv2 to fall in you windows, but dont know let me know what case may cause problems ?

gouessej · November 10, 2009, 10:01am

The threading of my own engine was strange, I did not need to deactivate DirectDraw on lots of machine under Vista whereas I have used OpenGL for years. Sometimes, the mouse cursor stayed frozen a few milliseconds, look at the TUER thread, we solved this problem in 2007.

In Ardor3D, its architecture does not rely on a delta, it uses the MouseEvent instances. That’s why I will have to use Riven’s suggestion.

DzzD · November 10, 2009, 10:24am

I have trouble to understand what you say … nothing related to DD or openg here I use this class with both standar Java2D & opengl canvas it just worked nice for me

heu… seems that the implementation I posted follow what Riven’s mention in its post above…

Riven · November 10, 2009, 12:32pm

What I suggested, is exactly the same as what DzzD suggests.

The difference in the implementation would be that I take the center of the window, and he takes the center of the screen, which is obviously only reliable in fullscreen mode.

gouessej · November 10, 2009, 12:44pm

Yes, I know, but it did not work as expected in my case, I have never really understand the exact reason why this approach did not work, I made some hypothesis.

You’re right, sorry.

I’m going to get the queue with Toolkit.getDefaultToolkit().getSystemEventQueue();
Then, I can use peekEvent(MouseEvent.MOUSE_MOVED) to get the newly created event.
If it is a centering event, I call getNextEvent.
It might be slow

Edit.: cylab told me once that we could directly use the queue. Now I know what he meant.

Markus_Persson · November 10, 2009, 5:54pm

You don’t need to skip an event.

gouessej · November 10, 2009, 10:22pm

What do you mean exactly?

Nate · November 11, 2009, 2:08am

FWIW, on Windows you can check a key event to see if it is injected. Eg, LLKHF_INJECTED. I don’t believe this is exposed from Java though.

Markus_Persson · November 11, 2009, 1:39pm

public void mouseMoved(MouseEvent mouseEvent) {
    int dx = mouseEvent.getX()-screenCenterX;
    int dy = mouseEvent.getY()-screenCenterY;
    if (dx!=0 || dy!=0) {
        handleMouseMove(dx, dy);
        robot.moveMouse(screenCenterX, screenCenterY);
    }
}

gouessej · November 11, 2009, 3:38pm

Thanks Markus_Persson. Now I do this, I have some things to improve to follow your advice:

public synchronized void mouseMoved(final MouseEvent e) {
        final int dx = e.getX() - _component.getWidth() / 2;
        final int dy = e.getY() - _component.getHeight() / 2;
        if (dx != 0 || dy != 0) {
            _clickArmed.clear();
            _clicks.clear();

            final MouseEvent me = new MouseEvent((Component) e.getSource(), e.getID(), e.getWhen(), e.getModifiers(), e
                    .getX(), e.getY(), e.getXOnScreen(), e.getYOnScreen(), e.getClickCount(), e.isPopupTrigger(), e
                    .getButton());

            initState(me);

            addNewState(me, _lastState.getButtonStates(), null);
            if (robot == null) {
                try {
                    robot = new Robot();
                } catch (final AWTException ex) {
                    ex.printStackTrace();
                }
            }
            final Point componentCenter = new Point(_component.getWidth() / 2, _component.getHeight() / 2);
            SwingUtilities.convertPointToScreen(componentCenter, _component);
            robot.mouseMove(componentCenter.x, componentCenter.y);
        }
    }

private void initState(final MouseEvent mouseEvent) {
        if (_lastState == null) {
            final int height = (_frame != null && _frame.getComponentCount() > 0) ? _frame.getComponent(0).getHeight()
                    : _component.getHeight();
            _lastState = new MouseState(mouseEvent.getX(), height - mouseEvent.getY(), 0, 0, 0, null, null);
        }
    }

    private void addNewState(final MouseEvent mouseEvent, final EnumMap<MouseButton, ButtonState> enumMap,
            final Multiset<MouseButton> clicks) {
        // changing the y value, since for AWT, y = 0 at the top of the screen
        final int height = (_frame != null && _frame.getComponentCount() > 0) ? _frame.getComponent(0).getHeight()
                : _component.getHeight();
        final int fixedY = height - mouseEvent.getY();

        final MouseState newState = new MouseState(mouseEvent.getX(), fixedY, mouseEvent.getX() - _lastState.getX(),
                fixedY - _lastState.getY(), (mouseEvent instanceof MouseWheelEvent ? ((MouseWheelEvent) mouseEvent)
                        .getWheelRotation() : 0), enumMap, clicks);

        _upcomingEvents.add(newState);
        _lastState = newState;
    }

gouessej · November 11, 2009, 5:12pm

Somebody suggested this:

_logicalLayer.registerTrigger(new InputTrigger(new MouseButtonReleasedCondition(MouseButton.LEFT),
                new TriggerAction() {
                    public void perform(final Canvas canvas, final TwoInputStates inputState, final double tpf) {
                        _cursorLock.x = -1;
                        _cursorLock.y = -1;
                        _isGrabbed = false;
                        _canvas.setCursor(null);
                    }
                }));

        _logicalLayer.registerTrigger(new InputTrigger(new MouseMovedCondition(), new TriggerAction() {
            public void perform(final Canvas canvas, final TwoInputStates inputState, final double tpf) {
                int x, y;
                x = inputState.getCurrent().getMouseState().getX();
                y = inputState.getCurrent().getMouseState().getY();

                if (_cursorLock.x == x && _cursorLock.y == y) {
                    _last.x = x;
                    _last.y = y;
                    return;
                }

                if (_isGrabbed) {
                    _uiRobot.mouseMove(_cursorLock.x, _canvas.getHeight() - _cursorLock.y);

                    final int dx = _last.x - x;
                    final int dy = _last.y - y;

                    _controlHandle.rotate(canvas.getCanvasRenderer().getCamera(), -dx, -dy);
                }
                _last.x = x;
                _last.y = y;
            }
        }));

        _logicalLayer.registerTrigger(new InputTrigger(new MouseButtonPressedCondition(MouseButton.LEFT),
                new TriggerAction() {
                    public void perform(final Canvas canvas, final TwoInputStates inputState, final double tpf) {
                        _canvas.setCursor(s_transparentCursor);
                        _isGrabbed = true;
                        _cursorLock.x = inputState.getCurrent().getMouseState().getX();
                        _cursorLock.y = inputState.getCurrent().getMouseState().getY();
                    }
                }));

and it works, it only needs some refactoring ;D

Markus_Persson · November 12, 2009, 2:59pm

Sometimes less is more.