Skip to main content

Java control OS mouse/keyboard

8 replies [Last post]
ncy
Offline
Joined: 2007-03-10

hi,

i was wondering how i would go about creating a Java program that is able to grab control of the mouse cursor and keyboard, and for example, have it click open and close windows, type some words in textpad or a URL location in a browser, browse around a bit, and close and shutdown the computer --- as if a person was actually using the mouse and keyboard, but all done automatically through the program code. so in effect, running the Java program would be like watching a pre-recorded video of somebody doing all this stuff. i suppose this has potential as a prank-type of program, but that's not my intention; i'm looking for a way to make a Java program that can automate tasks on this level, of mouse clicking and dragging and keyboard input. it'd be cool if somebody could also elaborate on grabbing control and processing other low-level input data, but mouse and keyboard are my main thoughts right now.

any ideas? btw, i have a Windows XP workstation.

thanx for any help and replies,

-n

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
ncy
Offline
Joined: 2007-03-10

thank you very much for your help! this gives me some direction and things to try :)

twalljava
Offline
Joined: 2004-07-26

The main problem with using java.awt.Robot on native applications is that generally you can't get much more information than just the window bounds. The only way to click on buttons or other widgets is to hard-code their absolute positions (which is feasible in a canned demo, not so much for a generic shortcut-recording mechanism).

JNA (http://jna.dev.java.net) has a demo included which does cross-platform shaped windows, which you could potentially use to decorate the desktop (although a java full-screen window with a robot snapshot might be better since you'd have better control of window layering).

ncy
Offline
Joined: 2007-03-10

thanx for the input, that's a good link.

tarbo
Offline
Joined: 2006-12-18

Not sure about actually seeing the cursor move, but I suppose you can get a good load of aid from the java.awt.Robot class.

ncy
Offline
Joined: 2007-03-10

nice, thanx for the quick reply. didn't know about java.awt.Robot before. any links to recommended sample code to download? the screencapture function looks pretty useful as well.

for seeing the cursor move, is there a way to draw an overlay on whatever is currently displayed on the screen, regardless of what application program currently has focus? kind of like a markup type of thing, where you can draw circles or whatnot or arbitrary graphics on the screen, as coded by the Java program. in this way, you'd be able to draw a persistent non-disappearing fake cursor based on the reported mouse position of the Robot class ... ideas?

tarbo
Offline
Joined: 2006-12-18

Sorry, other than googling on "Java tutorial java.awt.Robot", can't really help there. Never used the class before. It doesn't look difficult, though.

But it looks like (from the API doc) there's no need to bother with drawing your own cursor: it moves the system pointer around rather than generating events that your components react to.

Try something like this for a test:

[code]
public static void main(String[] args) throws AWTException {
Robot rob = new Robot();
rob.setAutoDelay(500); // 0.5s
for (int i = 0; i < 50; i++) {
rob.mouseMove(i * 5, i * 2);
}
}
[/code]

ncy
Offline
Joined: 2007-03-10

thanx again for the speedy reply. do you know about that graphics overlay thing? i.e. can i draw an arbitrary "Hello World" in a paint-brush like fashion (done in the code) straight over the current display, regardless of what program has focus?

tarbo
Offline
Joined: 2006-12-18

Hum... doubtful. A few things you can try, in personal order of preference, are:

[b]Transparent frames[/b] I'm not sure whether this is at all possible--never tried it--but perhaps you could make a frame whose panes (content, root) are all transparent. Again, not sure whether it's at all possible, and you [i]may[/i] need to use AWT components rather than Swing to pull it off.

[b]OS-specific "Overlay"[/b] It should be possible to manhandle DirectX or OpenGL (or your window manager) into giving you a transparent overlay to draw on. I'm not sure how you would go about this via Java, though, and you will need to install the specific extensions (probably Java3D). Note that portability may take a hit here.

[b]Static background[/b] Get a snapshot of the whole desktop, create a fullscreen window, draw the background, and go from there. You should have a snapshot of the desktop that you can now draw on at your leisure. Of course, it's static, and won't show such things as dynamic images unless you draw them yourself.