Using java.lang.reflect to Create a Quake-Style Debug Console?

[eraboin]

I’ve been experimenting with Java 5’s new annotation feature, which allows you to add metadata to your code and refer to it during runtime. Combining this with java.lang.reflect you can essentially sweep through your program and create an annotated index of any public Class, Field and Method objects. You can also get a String representation of their name, allowing you to place them in a Hashtable, or some other data structure for future reference. Finally (and almost frighteningly) you can change fields or invoke methods at will, or even create new instances of certain classes.

I’ve only recently learned how to use these techniques, and they have a sort of ‘black magic’ feel to them. They seem like they’re incredibly powerful if you use them the right way, but could also cause serious problems if you apply them indiscriminately. These techniques are also very costly, and shouldn’t be relied upon in code which needs to execute quickly. Fortunately, they’re all well documented in the Java API, and Java is kind enough to throw Exceptions when we do the wrong thing.

Consider the following code;

import java.lang.reflect.*;

public class SomeClass
{
	public int variable_1 = 1;
	public int variable_2 = 2;
	public String hello = "hello";

	public static void main(String[] args)
	{
		SomeClass myClass = new SomeClass();

		String name, value;

		for(Field field : myClass.getClass().getDeclaredFields())
		{
			if(Modifier.isPublic(field.getModifiers()))
			{
				name = field.toGenericString();

				try
				{
					value = field.get(myClass).toString();
					System.out.println(name + " = " + value);
				}
				catch(IllegalAccessException e)
				{
					System.out.println("The field " + name + " is not accessible.");
				}
			}
		}
	}
}

The output should be;

public int SomeClass.variable_1 = 1
public int SomeClass.variable_2 = 2
public java.lang.String SomeClass.hello = hello

Which means we can access actual Field objects during runtime, which give us a String representation of their name. Be aware that if the fields are private, and they’re stored in some other class (ie; not the one we’re currently in) then that’s a reason for the code to throw an IllegalAccessException. We’re already detecting this with the call to Modifier.isPublic(), but the try/catch clause is just there because Java enforces it. It’s good to have decent exception handling anyway.

You can also modify fields in the following manner;

import java.lang.reflect.*;

public class SomeClass
{
	public int variable_1 = 1;
	public int variable_2 = 2;
	public String hello = "hello";

	public static void main(String[] args)
	{
		SomeClass myClass = new SomeClass();

		System.out.println(myClass.variable_1);

		try
		{
			Field myField = myClass.getClass().getDeclaredField("variable_1");
			myField.set(myClass, 1234);
		}
		catch(NoSuchFieldException e)
		{
			System.out.println("variable_1 does not exist");
		}
		catch(IllegalAccessException e)
		{
			System.out.println("variable_1 is not accessible");
		}

		System.out.println(myClass.variable_1);
	}
}

Which should output;

1
1234

Meaning we can refer to fields by their String representation and then modify their values. This way we don’t have to set up any bizzarre data structure when we define our classes, and all the hard work is shifted to the code which is actually doing the modification. The code doesn’t have to be written in each of the classes either; you can create one class which is responsible for doing all the indexing, and pass it Object references for the classes you want to index. Someone could change the code for the class we’re looking at, add or remove fields, and they could be indexed in the same manner.

The only catch is that the programmer should be aware that their code is being watched, because changes to public fields could be made unpredictably. You could avoid this by indexing methods instead, and having the programmer write get/set methods for all their fields which filter invalid input. But, indexing fields is easier, and you’re supposed to account for this unpredictability in public fields anyway.

Finally, using annotations, which are new to Java 5, let you conveniently add metadata to your code. You can define a runtime-accesible Annotation in the following manner;

import java.lang.annotation.*;

@Retention(RetentionPolicy.RUNTIME)
public @interface SomeAnnotation
{
	String someName();
	double someValue();
}

This should be stored in it’s own *.java file, but it doesn’t behave like a regular interface. It should be applied to fields in the following manner;

import java.lang.reflect.*;

public class SomeClass
{
	@SomeAnnotation(someString="This is variable 1", someValue=3.14159265)
	public int variable_1 = 1;

	@SomeAnnotation(someString="This is variable 2", someValue=2.71828183)
	public int variable_2 = 2;

	@SomeAnnotation(someString="Hello World", someValue=42)
	public String hello = "hello";

	public static void main(String[] args)
	{
		SomeClass myClass = new SomeClass();

		String name, value;

		for(Field field : myClass.getClass().getDeclaredFields())
		{
			if(Modifier.isPublic(field.getModifiers()))
			{
				System.out.println("Metadata for " + field.toGenericString() + ":");

				SomeAnnotation annotation = field.getAnnotation(SomeAnnotation.class);

				if(annotation!=null)
				{
					System.out.println(annotation.someString());
					System.out.println(annotation.someValue());
				}

				System.out.println();
			}
		}
	}
}

Which should output;

[i]Metadata for public int SomeClass.variable_1:
This is variable 1
3.14159265

Metadata for public int SomeClass.variable_2:
This is variable 2
2.71828183

Metadata for public java.lang.String SomeClass.hello:
Hello World
42.0[/i]

You could use these annotations to specify legal values for particular fields (for instance, a minimum and a maximum) and then, when your Console indexes them, it knows what input to allow from the user. You can also annotate methods and classes, and use the same approach to indexing them as well.

Now, what I don’t know, is how reasonable this approach is for a larger project. I’m attempting to implement it in the game I’m working on now. The game has a lot of configuration settings, particularly for the graphics, which I may want to modify during runtime. I mentioned a ‘Quake-style’ console because I hope to successfully emulate the console behavior in certain first person shooters. I realize that the consoles for those games were probably not managed in quite the same way, but I don’t see why the behavior should be any different.

Now… it’s true that these techniques are costly, but I figure if I index everything I’m going to change ahead of time, and access them only when I want to modify/retrieve values, then it can’t be that bad. The console would not be executing commands at 60 frames per second… a person can’t even type that quickly. So despite the fact that the game itself needs to run smoothly, I’m not sure the console does. Anything which requires speed I can write special methods for.

More information about java.lang.reflect can be found here: http://javaalmanac.com/egs/java.lang.reflect/pkg.html
More information about annotations can be found here: http://www.onjava.com/pub/a/onjava/2004/04/21/declarative.html

Thoughts and comments appreciated. Criticisms feared but accepted…

…

ergh… this was supposed to go to the Shared Code board… oh well.

[swpalmer]

You’ve discovered reflection and applied it to come up with some of the same ideas as Bean Shell (Google it if you aren’t familiar). The advantage that you’ve hit on is the use of annotations to enhance the concept. I think it is actually a good idea for the in-game console, such as that used by id.

[CaptainJester]

@swpalmer

You’ve discovered reflection and applied it to come up with some of the same ideas as Bean Shell (Google it if you aren’t familiar). The advantage that you’ve hit on is the use of annotations to enhance the concept. I think it is actually a good idea for the in-game console, such as that used by id.

I agree.

As you said, the console does not have to update quickly. Also, you will only execute that code after a console command is entered, so the slow part of the code will only be entered when necessary and will be short lived.

ps. Great idea.

[eraboin]

I’ve set up a simple version of it. It’s actually pretty fast. All you do is call a method named “addFields(obj)” and it checks for all annotated fields within the object, then stores information about them as a command.

I created annotations for all primitive types (boolean, int etc…) and Strings. The numerical values can specify a min, max, and an array of restricted values. For strings you can specify a regular expression. All annotations tell whether or not a field is editable, and can return help text.

The actual commands are stored as a Command object, which just has an execute(string[]) method. Once the commands have been created it doesn’t take much longer to execute a field command than it does an anonymous one. The worst it has to do is call field.get(parent), or field.set(parent, arg).

The only catch is, each command keeps a reference to the object that contains the field, so the object will never be cleaned up by the garbage collector until the command is destroyed as well. I suppose I could create a “removeFields(obj)” method that compares the object with each field command and then removes the ones that match… but that would need to be a conscious step.

[blahblahblahh]

@eraboin

I’ve set up a simple version of it. It’s actually pretty fast. All you do is call a method named “addFields(obj)” and it checks for all annotated fields within the object, then stores information about them as a command.

I created annotations for all primitive types (boolean, int etc…) and Strings. The numerical values can specify a min, max, and an array of restricted values. For strings you can specify a regular expression. All annotations tell whether or not a field is editable, and can return help text.

So, basically, you’ve created a java OOP interface that enforces limits and provides information about them, and … uh, no, you haven’t. Which is where I’m confused - what’s the point of doing this? You still have to recompile the code on changes, and you’ve got a weaker interface than OOP would give you - so you seem to have the worst of both mainstream java (the lang) and [insert favourite scripting language here].

I’m not complaining, I’m just saying I don’t “get” it, and wondering what I’m missing?

[eraboin]

Well… look, this is how you’d define a Command normally;

Console console = new Console();

console.addCommand("name", new Command()
{
	public String execute(String[] args)
	{
		doSomething();
		return "I did something";
	}
});

…and you can still do that.

If you wanted to you could write a Command for each Field you want to index. The only thing my code does is generate the Commands automatically based on the Field’s annotations. There really isn’t any difference between the two methods, it’s just that one requires writing a lot of extra code for each Field, and the other you can just pass in an Object.

This is what some of the annotation looks like;

   /** This is a Double */
   @DoubleField(help = "non-zero values", restricted = {0})
   public double nonZero;

   /** This is a Boolean */
   @BooleanField(editable = false)
   public boolean noEdit;

   /** This is a String */
   @StringField(help = "at most 8 characters", max=8)
   public String someString = "someText";

   /** This is a binary String */
   @StringField(pattern = "[0*1*]*")
   public String binaryString = "01010";

The different parameters are optional… you can include some, not others. It’s no more than javadoc would ask for, except now you can modify it from a console without writing a lot of extra code to do it.

The Console also allows you to create your own Field annotations (supposing you don’t want to use primitives) by using a meta-annotation that specifies some Command to deal with it. You can also prefix all the Fields from a specific Object before you load them into the console, by using addFields(“prefix”, obj).