Thread: DirectRobot - A fast Java Robot that allows for screen recording

I actually wrote this for my RDP (Remote Desktop Program) that I was writing and I thought I'd share it with ya'll so you can do something fun with it.

Why did I make this? Because java.awt.Robot is very indirect, has a lot of added garbage esp. when dealing with screen captures. Specifically to use the screen capture method for an RDP would be a nightmare, everytime you call the method it first has to create a pixel array to call the method to grab the screen pixels, then it has to make it into a BufferedImage which then requires another pixels array, then finally when you get your BufferedImage you have to have another pixel array to grab the data from it, in other words: nightmare.

This can cause very slow speeds on some machines, and in no doubt it will cause slower performance on just about any machine. In 2004 someone submitted a bug report about it, 8 years later after being accepted nothing has been done.

Bug ID: 5090776 Need more efficient screen capture.

I originally discovered these methods a few years ago, never really put them to use for anything, but now that I am writing an RDP I really needed them so I went further than just Windows support and asked around and got Windows, Linux, AND Mac 1.5/1.6/1.7 support. NOTICE: java.awt.Robot was not added until 1.5 so there is no reason to have anything earlier unless you want to write your own natives, then you probably aren't even interested in this..

Yeah!

Thanks to those who helping me get the dumps for the other platform methods.

Danny (Clienthax) - Linux
Daniel (Mister Maggot) - Mac 1.5 (or was it 1.6, idk)
Avatar Realms (thanks to Danny for referring me) - Mac 1.6/1.7

I also decided to include a more direct way of grabbing the mouse information, that doesn't involve the creation of a new object every single time...

If anyone finds a platform that can't use the direct pixel grabbing method, please let me know and send me your dump.

NOTE: I did leave the debugging stuff in there, so if you don't want it I suggest you just remove it. But please if it doesn't work post your results!

And as always, have fun.

Code:

import java.awt.*;
import java.awt.peer.*;
import sun.awt.*;
import java.lang.reflect.*;
import java.net.*;
import java.io.*;
import java.util.*;

public final class DirectRobot
{
	public DirectRobot() throws AWTException
	{
		this(null);
	}

	public DirectRobot(GraphicsDevice device) throws AWTException
	{
		if (device == null)
			device = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();

		this.device = device;
		Toolkit toolkit = Toolkit.getDefaultToolkit();
		peer = ((ComponentFactory) toolkit).createRobot(null, device);
		Class<?> peerClass = peer.getClass();
		Method method = null;
		int methodType = -1;
		Object methodParam = null;
		try
		{
			method = peerClass.getDeclaredMethod("getRGBPixels", new Class<?>[] { Integer.TYPE, Integer.TYPE, Integer.TYPE, Integer.TYPE, int[].class });
			methodType = 0;
		}
		catch (Exception ex) { }
		if (methodType < 0)
			try
			{
				method = peerClass.getDeclaredMethod("getScreenPixels", new Class<?>[] { Rectangle.class, int[].class });
				methodType = 1;
			}
			catch (Exception ex) { }

		if (methodType < 0)
			try
			{
				method = peerClass.getDeclaredMethod("getScreenPixels", new Class<?>[] { Integer.TYPE, Rectangle.class, int[].class });
				methodType = 2;
				GraphicsDevice[] devices = GraphicsEnvironment.getLocalGraphicsEnvironment().
getScreenDevices();
				int count = devices.length;
				for (int i = 0; i != count; ++i)
					if (device.equals(devices[i]))
					{
						methodParam = Integer.valueOf(i);
						break;
					}

			}
			catch (Exception ex) { }

		if (methodType < 0)
			try
			{
				method = peerClass.getDeclaredMethod("getRGBPixelsImpl", new Class<?>[] { Class.forName("sun.awt.X11GraphicsConfig"), Integer.TYPE, Integer.TYPE, Integer.TYPE, Integer.TYPE, int[].class });
				methodType = 3;
				Field field = peerClass.getDeclaredField("xgc");
				try
				{
					field.setAccessible(true);
					methodParam = field.get(peer);
				}
				finally
				{
					field.setAccessible(false);
				}
			}
			catch (Exception ex) { }

		if (methodType >= 0 && method != null && (methodType <= 1 || methodParam != null))
		{
			getRGBPixelsMethod = method;
			getRGBPixelsMethodType = methodType;
			getRGBPixelsMethodParam = methodParam;
		}
		else
		{
			System.out.println("WARNING: Failed to acquire direct method for grabbing pixels, please post this on the main thread!");
			System.out.println();
			System.out.println(peer.getClass().getName());
			System.out.println();
			try
			{
				Method[] methods = peer.getClass().getDeclaredMethods();
				for (Method method1 : methods)
					System.out.println(method1);

			}
			catch (Exception ex) { }
			System.out.println();
		}
	}

	public static GraphicsDevice getMouseInfo(Point point)
	{
		if (!hasMouseInfoPeer)
		{
			hasMouseInfoPeer = true;
			try
			{
				Toolkit toolkit = Toolkit.getDefaultToolkit();
				Method method = toolkit.getClass().getDeclaredMethod("getMouseInfoPeer", new Class<?>[0]);
				try
				{
					method.setAccessible(true);
					mouseInfoPeer = (MouseInfoPeer) method.invoke(toolkit, new Object[0]);
				}
				finally
				{
					method.setAccessible(false);
				}
			}
			catch (Exception ex) { }
		}
		if (mouseInfoPeer != null)
		{
			int device = mouseInfoPeer.fillPointWithCoords(point != null ? point:new Point());
			GraphicsDevice[] devices = GraphicsEnvironment.getLocalGraphicsEnvironment().
getScreenDevices();
			return devices[device];
		}
		PointerInfo info = MouseInfo.getPointerInfo();
		if (point != null)
		{
			Point location = info.getLocation();
			point.x = location.x;
			point.y = location.y;
		}
		return info.getDevice();
	}

	public static int getNumberOfMouseButtons()
	{
		return MouseInfo.getNumberOfButtons();
	}

	public static GraphicsDevice getDefaultScreenDevice()
	{
		return GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice();
	}

	public static GraphicsDevice getScreenDevice()
	{
		return getMouseInfo(null);
	}

	public void mouseMove(int x, int y)
	{
		peer.mouseMove(x, y);
	}

	public void mousePress(int buttons)
	{
		peer.mousePress(buttons);
	}

	public void mouseRelease(int buttons)
	{
		peer.mouseRelease(buttons);
	}

	public void mouseWheel(int wheelAmt)
	{
		peer.mouseWheel(wheelAmt);
	}

	public void keyPress(int keycode)
	{
		peer.keyPress(keycode);
	}

	public void keyRelease(int keycode)
	{
		peer.keyRelease(keycode);
	}

	public int getRGBPixel(int x, int y)
	{
		return peer.getRGBPixel(x, y);
	}

	public int[] getRGBPixels(Rectangle bounds)
	{
		return peer.getRGBPixels(bounds);
	}

	public boolean getRGBPixels(int x, int y, int width, int height, int[] pixels)
	{
		if (getRGBPixelsMethod != null)
			try
			{
				if (getRGBPixelsMethodType == 0)
					getRGBPixelsMethod.invoke(peer, new Object[] { Integer.valueOf(x), Integer.valueOf(y), Integer.valueOf(width), Integer.valueOf(height), pixels });
				else if (getRGBPixelsMethodType == 1)
					getRGBPixelsMethod.invoke(peer, new Object[] { new Rectangle(x, y, width, height), pixels });
				else if (getRGBPixelsMethodType == 2)
					getRGBPixelsMethod.invoke(peer, new Object[] { getRGBPixelsMethodParam, new Rectangle(x, y, width, height), pixels });
				else
					getRGBPixelsMethod.invoke(peer, new Object[] { getRGBPixelsMethodParam, Integer.valueOf(x), Integer.valueOf(y), Integer.valueOf(width), Integer.valueOf(height), pixels });

				return true;
			}
			catch (Exception ex) { }

		int[] tmp = getRGBPixels(new Rectangle(x, y, width, height));
		System.arraycopy(tmp, 0, pixels, 0, width * height);
		return false;
	}

	public void dispose()
	{
		getRGBPixelsMethodParam = null;
		Method method = getRGBPixelsMethod;
		if (method != null)
		{
			getRGBPixelsMethod = null;
			try
			{
				method.setAccessible(false);
			}
			catch (Exception ex) { }
		}
		//Using reflection now because of some peers not having ANY support at all (1.5)
		try
		{
			peer.getClass().getDeclaredMethod("dispose", new Class<?>[0]).invoke(peer, new Class<?>[0]);
		}
		catch (Exception ex) { }
	}

	protected void finalize() throws Throwable
	{
		try
		{
			dispose();
		}
		finally
		{
			super.finalize();
		}
	}

	private Object getRGBPixelsMethodParam;
	private int getRGBPixelsMethodType;
	public final GraphicsDevice device;
	private Method getRGBPixelsMethod;
	private final RobotPeer peer;
	private static boolean hasMouseInfoPeer;
	private static MouseInfoPeer mouseInfoPeer;
}

Still need a BufferedImage?

Use this, trust me there is no copying involved here, you change a pixel in the array and the image changes too, try it! (this is what Jagex uses for their custom pixel rendering)

Code:

ColorModel model = new DirectColorModel(32, 0xff0000, 0xff00, 0xff, 0xff000000);
BufferedImage image = new BufferedImage(model, Raster.createWritableRaster(model.createCompatibleSampleModel(width, height), new DataBufferInt(pixels, width * height), null), false, new Hashtable<Object, Object>());

Remember though, if your pixels are JUST RGB and not ARGB (alpha + rgb), then you need to remove "0xff000000" from the DirectColorModel arguments, just remove it. But the Robot should return ARGB, if not let me know. But chances are if you are using this for something else and you do not know what ARGB means, then you have RGB.