I’ve been looking for different ways to send keystrokes, mouse movements and clicks to GUI applications. Under Windows, this is straight forward with AutoIt. The BASIC-like language allows you to read files, start applications, send keystrokes move the mouse etc. A nice feature is to wait for certain dialogs, for example:
; wait for tightvnc window
WinWaitActive("New TightVNC Connection")
; send hostname and port number
Send($hostname & ":" & $displaynumber)
This makes sure that the keystrokes actually end up in the right window. Mouse coordinates can be specified relative to the application window. Using these features allows you to write ‘reasonably’ robust code. But don’t expect your AutoIt script to run flawlessly on other PC’s. There are a lot of dependencies, like dialogs that look different in another version. Internationalization might cause the window title to read “Nieuwe TightVNC Verbinding” (Dutch) and the above code snippet will fail. Even user preferences like font sizes, window themes and screen resolution might result in unexpected behavior.
Anyway, it’s just fun to figure out which sequences to send to an application and let it process a batch of documents!
Ok, this works pretty nice under windows. But what about my favorite platform?
I thought it couldn’t be difficult manipulating mouse clicks and keystrokes with the Penguin…
Well, the most difficult task was collecting the required tools! I ended up with vnc and rfbplaymacro.
Let me explain 
. VNC uses the RFB (Remote FrameBuffer) protocol to remotely control another computer. While vncviewer (vnc client) is an interactive application (you interact with the remote computer as if you where sitting behind it), rfbplaymacro is ‘headless’. It allows you to send over the keystrokes and mouse clicks, but it doesn’t show you what happens. Instead of actually using your mouse and keyboard, rfbplaymacro expects a macro fed into stdin. The macro is a plain text list of mouse and keyboard events. The demo macro that is supplied with the rfbplaymacro source code displays an xclock for 10 seconds:
RFM 001.000 password "mypasswd123" pointer 100 20 delay 1s type "xclock -geometry +400+200 &" press Return type "xkill" press Return pointer 410 210 delay 10s click 0
The first line tells the parser which of the possible macro versions is used. I’ve only seen just one version. The password statement is optional, it’s also possible to specify the password when playing the macro, see the next code snippet. This script assumes the remote desktop runs an xterminal in the upper left of the screen. By moving the mouse to the upper left (pointer 100 20), the xterm gets the input focus (depending on the window manager). After the mouse movement, the macro pauses for one second, specified by delay. Then the xclock is started in the background, by typing the xclock command and hitting return. Xkill is fired up in the same fashion. The mouse pointer is transformed in a deadly weapon! Another pointer instruction moves the cursor to xclock. The dangling ‘Sword of Damocles` is hanging for the full 10 seconds before final and deadly mouse click is sent.
To play back this rather useless, but illustrating macro, use:
$ rfbplaymacro --server=linuxbox:9 < test.rfm
This will connect to a running vncserver (desktop :9) on ‘linuxbox’ and sends over the mouse and keyboard events specified in the marcro test.rfm. Connect a vncviewer to the same desktop and you will see it happen!