Scripting language embedding in Java

[blahblahblahh]

I often want to embed a simple scripting language in games, but it’s kind of hard finding a really good scripting language. Has anyone here actually used some in a real project? (suggestions of things that people have seen but haven’t used themselves are 2-a-penny, and unfortunately there are SO many out there that it’s an impossible task to compare all of them!)

To date, I’ve nearly always rolled my own (putting those 6 months of Compiler Construction lectures to good use…). It’s pretty easy to write a new language once you know all about compiler design, and with the automatic parser-generators (yacc etc) it can be quicker to create a new language, implement it, and use it than to learn an existing one and embed it.

But I always feel guilty about making other people learn an extra language :), and wish I could just embed something standard. Problem is, most available scripting languages either suck, or are really good but hard to learn (e.g. SmallTalk).

[blahblahblahh]

[quote]Has anyone here actually used some in a real project?
[/quote]
P.S. I hate it as a language, but my favourite scripting language from non-games projects is LotusScript/VBScript/etc…it’s really easy for casual/non-programmers to edit and use. It’s also quite hard to break the source code by editing it.

The only real problem I have with it is that the lack of imposed / encouraged structure means novice programmers quickly generate unmanageable scripts :(.

[Markus_Persson]

http://www.beanshell.org/

It kicks ass. =)

[tortoise]

I couldn’t be more happy with Jython. The more I use it in my engine, the more I love it. It’s a lot faster than BeanShell too

[blahblahblahh]

I accidentally found this:

http://www.javaworld.com/javaworld/jw-04-2002/jw-0405-scripts.html

Which is a nice starting point, but very short on detail.

EDIT: Actually, the author is either lazy or dumb. He chose for his “example of writing code in the scripting languages” an example that is almost identical to write in java as in the langs…he didn’t even implement the same code in the different languages, AFAICS. It would be much better with an example of the kind of thing you actually would WANT to use a scripting lang for :(.

I must point out that in the benchmarking the article author apparently didn’t read the basic intro docs for beanshell - they make it clear that beanshell generates a lot of objects when doing something like iterating through a massive array; essentially, it currently never uses basic datatypes, so I suspect the performance is actually normally much better, unless you are doing massive loops (which is unusual anyway.

[blahblahblahh]

[quote]I couldn’t be more happy with Jython. The more I use it in my engine, the more I love it. It’s a lot faster than BeanShell too
[/quote]
OK, but WHY? (let’s forget exec speed differences for the time being; I think most people aren’t too bothered by speed if they’ve decided to use a scripting language anyway )

Can you give some examples of how you use Jython instead of Java, and why you think Jython works so well there instead of beanshell (sounds like you’ve used BS a bit too?)

[blahblahblahh]

And I eventually found this:

http://www.robert-tolksdorf.de/vmlanguages

…which has “about 173” languages written on the JVM, so generally you can embed them within a java app quite easily. Unless you’re a student of programming languages, you might not recognise the major groups (which are implementations of other well known langauges in java), but near the bottom are lots of weird and wonderful “extensions” to java.

Anyone care to review them all ? ;D

[nonnus29]

I followed your approach and simply rolled my own. I started with some opcodes and a virtual machine then wrote a nasty little BASIC interpreter for it. I felt really overwhelmed by the sheer complexity of building up a game with scripting and alot of features so I put it all on the back burner and started playing around with 2d and 3d software rendering. I should really get back to that though…

I don’t think its too much to ask users to learn a new language. After you learn a couple it becomes easy.

To me the epitome of a game scripting language is the Half Life SDK. Its completely oo and introduces state as a basic type.

But you wanted 1st hand experience so …

[sma]

[quote]Anyone care to review them all ?
[/quote]
Sure. No problem. I’ll give it a try.

First section, precompilers, not suitable for scripting, -6.

Second section, tcl, a scripting language famous for its UI toolkit “tk”. Both links refer to the same implementation, -1. Quite usable, if you like tcl (I don’t). +1

Third section, functional programming, all implementation not suitable, -5.

Next section, Lisp and co, contains some (IMHO) interesting candidates - if you like Lisp (I do). I don’t know Jatha which seems to be quite new. Documentation link on SF page gives a 404 though. The big four are Bigloo, SISC, Kawa and JScheme - most other links are unfinished or experimental, -11. Bigloo just generates Java (or .NET) bytecode but isn’t written in Java so it’s not really suitable as a scripting language. JScheme once started as SILK (Scheme in less than 50k) and is still small and therefore probably an interesting candidate. But it’s slow. Kawa compiled Scheme source dynamically to class files. SISC is still an interpreter, but a more sophisticated one and faster than JScheme (but kawa is still faster) - and it come with a large lib. However, IMHO for a good integration into a Java application, not the completeness of the Scheme implementation is important but size, performance and extensebility. As I’d always prefer a true OO languages, I’d choose neither implementation. +3

Basic, next section, boring. One link is a converter, another link gives a 404, I know of no Basic suitable to embed into Java Application. Something VBA/VBScript compatible would be an interesting project, though. -8.

Starlogo is really nice and a powerful language, but not embeddable. I never really tried the other 4 implementations but checked them out before I tried to create a tiny logo for fun. So I don’t think, that one of the links is suitable as an embedable scripting language. -5. And, there’re better languages than logo. As you might know, logo has much in common with Lisp/Scheme - just a different syntax. A very interesting language with a similar syntax is Rebol - not available for Java though - I once tried to create a Rebol interpreter but lost interest when I eventually started to dislike aspects of that language.

Next section, logic progamming. Postly prolog. Very suitable for embedding (at least a few implementations listed here, I once tried W-prolog) but not really what you’d call a scripting langague - too strange, too different is the declarative paradigm from the imperative paradigm. Quite a few links point languages not written in Java, JESS is an expert system shell, -16.

I can’t say anything to Eiffel. But both projects look like they’re compilers not embeddable into a Java app. -2.

Smalltalk: Perhaps because that’s my favorite language, I put the bar very high here, but simply forget the first. Talk2 tried to rebuid a Smalltalk environment and the project never took off and it seems they try to hide it. Bistro is a bastard with IMHO got all the bad features from Smalltalk combined the even worse features from Java That’s not Smalltalk anymore. No project is embedable. -3

Various OO languages, only a few comments: Self is a really cool language (the Self research project took place from 1987 to 1995) and its VM was the base for the Java-Hotspot-VM. Some guys of the Self project continued to work on its (IMHO) ground breaking ideas and created Cecil - an even cooler language Self is a condensed, class-less, prototyp-based Smalltalk variant. Cecil combines the prototype-based approach with a tiny bit of functional programming and the powerful OO model of CLOS/Dylan. BTW, JavaScript a child of Self. As is NewtonScript, the scripting language for the Apple Newton computer, the first PDA. dSelf is a Self system created in Java as a Master Thesis (or PHD, I don’t remember) with remote - distributed - objects.

If you haven’t checked it out, give Nice a look. Created by some French guys (related to the OCaml project I think) this languages combines Java with much needed functional and OO features. Basically, it’s a crossbreed of Cecil and Java which lost the prototype though. They added a real static typsystem, multimethods and some other nice featues, keeping compatibility with Java.

But nothing to embed. -13. Let’s not talk about Ada or Cobol. -3.

Finally, section Scripting. Creating a scripting language based on XML is a violation of mind. A scripting language shall be easy to real for humans, but XML isn’t. It’s easily parsable for computers. A scripting language should also IMHO be at a hight abstraction level than the languages used to create the base components - otherwise you wouldn’t need an additional language. The only excuse might be that the base language isn’t dynamic. But’s that’s a problem of that language… JRuby would be my favorite but it seems that project is dying or died already. Otherwise, Ruby combines the best features of Smalltalk and (gasp) Perl with in a syntax which is IMHO very readable and even better than Python. It allows very high level abstractions thanks for blocks (aka closures), allows meta programming (everything is an object you can programm in the language) and still looks like the familiar Basic for the casual programmer. BSF, BTW, isn’t a scripting language at all but an API you can use in your application to make the scripting language plugable - similar to the Microsoft Scripting Host. I dislike C-like languages (it’s enough to have Java C-like) and I don’t think that a Forth-like are easy enough to use (although they’re interlectual fun to use and to create). Forget FESI.

From that most important section, the usable alternatives IMHO are: DynamicJava, Rhino, BeanShell, Jython and Pnuts.

Use the first, I you think, that using the same Java syntax as a scripting language is a good idea. Use Rhino if you think that JavaScript is a kind of standard (actually it is: ECMA262 and people might know this from Webbrowsers, Flash, etc. Use Jython if you like Python. And you Pnuts or BeanShell if you want a java-like, but slightly more dynamic language a lot of other people also use… Or use JScheme or Kawa.

I already used Pnuts and Rhino as embedded language and both work great. I don’t really like neither language. Pnuts is somehow boring and I used it mainly because it’s a very compact system, less than 100k and I couldn’t convince the customer to use Jscheme Rhino would be my favorite because of the standard-argument but I dislike that this language does this: “1” - “1” = 0, “1” + “1” = “11”. Overloading the “+” for strings in combination with JavaScripts automatic type conversion was IMHO a very bad idea. Jython is too big for my taste to embed and the language might be too powerful. I’d probably talk different here if Ruby - which I prefer over Python - would be available.

The remaining sections doesn’t list scripting/embedable languages. So there aren’t 173 languages but only 8.

For a long time now I’m on a quest for the (at least my) perfect scripting language. I checked out quite a few languages and tried to implemented interpreter for a few of them. The language needs a tiny core and must be extentable. It should be human reable (otherwise, I’d pick Scheme), should feature true OO, higher order functions and it would be nice especially for games if you could embed a tiny rules engine, oops, again Lisp). Besides Scheme, Smalltalk is a good candidate but still, Ruby has a more familar Syntax. I’d pick Dylan because of its more advanced OO, but I like neither the original Scheme-like Syntax nor the Pascal-like language Apple gave that language later. And I’d like to have an optional static type system for that language and this isn’t available (perhaps even possible) for Dylan. I’ve seen the static type systems for Haskell or Cecil and, well, I feel that no average programmer can ever learn this. I experimented also with clones or Rebol and Curl, with Self and Lua but every time, I found something I didn’t like and abandoned that language. Therefore I’ve half-finished implementations of Java-Interpreters for most languages I mentioned I wanted to recreate existing language to not have to create documentation/tutorials…

So if anybody wants to discuss pros and cons of languages or create one or two, I’m happy to help, advance, discuss this matter.

My goal for the scripting (besides usage in “commercial” programs) would be the game logic of a turn-based (web based) strategy game engine, think of PB(E)M games for exmple.

PS: It’s Smalltalk with a lowercase t. You don’t write JaVa, do you?

[sma]

[quote]Jython is faster than BeanShell. OK, but WHY?
[/quote]
Jython dynamically compiles its code to class files which are then loaded via a custom class loader and JITed by the VM. Kawa and Rhino do the same and are faster compared to SISC or FESI. At least the old 1.x version of BeanShell was a simple interpreter.

OTOH, compiling class files will not always speed up a scripting system because it takes quite a few millisecs to transform the internal AST into bytecode, generate a class file, load that, let the VM verify it and compile it. Code that is run only once might be actually slower.

[blahblahblahh]

Smalltalk: Perhaps because that’s my favorite language,

I’ve implemented a rule-based function despatcher for java that IIRC was similar to the function despatch of ST. Some of ST is definitely useful to have in a highly dynamic system (where OO isn’t good enough at isolating the code from architectural/design changes).

Various OO languages, only a few comments:

If you haven’t checked it out, give Nice a look. Created by some French guys (related to the OCaml project I think) this languages combines Java with much needed functional and OO features. Basically, it’s a crossbreed of Cecil and Java which lost the prototype though. They added a real static typsystem, multimethods and some other nice featues, keeping compatibility with Java.

Multi-methods: my biggest gripe with Java’s JLS these days. I’ve often argued that a true OO language ought to support multi-methods just to avoid breaking the principle of least surprise. In my perfect world, Java 1.5 would include it alongside autoboxing, generics, etc :).

[blahblahblahh]

My goal for the scripting (besides usage in “commercial” programs) would be the game logic of a turn-based (web based) strategy game engine, think of PB(E)M games for exmple.

Funny you should say that…I have to make a decision very soon on what scripting language will be used in a browser-based-game whose heritage is that it used to be a PBM. At the moment, it looks like we’ll probably go with BeanShell for two reasons. One, *Python is useless because of concerns over editing source inside HTML pages (tab? Oh damn!). It could be done, but only by violating the language, and I’m not prepared to support a bastardised Python. Two, quick integration-time and simplicity are more important than perfection for this game.

Long-term, though, I have a huge need for things not found in java, like multi-methods. Usually this is in isolated areas of a large system, so it’s acceptable to have big chunks of class hierarchy etc in a separate language subsumed inside a java environment. It’s even possible to run multiple incompatible interpreters side-by-side, but I’d much prefer to have everything inside JVM’s (note: I sometimes have multiple JVM’s per machine, e.g. to run “not quite trusted” interpreters or code; you have to be careful about the IPC overhead when talking JVM-to-JVM, as at the moment I just use the loopback interface, which fortunately is fast enough.).

Most of this game is written in Java, but that’s because most of it doesn’t need to evolve over time. The ultra dynamic stuff needs to go in soon, e.g. quests, and there’s no way we’ll be doing that in Java.

FYI for anyone wondering “should I use Java or [scripting language X]?” for a similar game, Java’s been a very good choice for most of it:

[] Very little added code. With really good OO design, you can quickly build a system where it takes almost the same amount of typing to achieve something in Java as in a scripting lang. And a bonus is that small bits of code can be replaced with different versions very easily, whilst retaining the original (because of class encapsulation).
[] Fast and efficient; lots of network-messages are pre-cached in long-lived ByteBuffers. Some are not modified for days at a time, but read from very frequently. With a scripting lang, you can of course stick a cache in front and achieve much the same effect. Although interacting perfectly with external caches isn’t trivial.
[*] type safety and exception handling. The exception handling has been the biggest win - when you want a server that has to run for weeks at a time with only limited admin intervention, it’s very re-assuring to know that every Throwable is caught at the most relevant points (e.g. the request-processing loop has a catch-all built in, so that if anything goes wrong on one request, subsequent requests are guaranteed not to be affected. Unless the problem is an out-of-memory error or smiilar :)).

[blahblahblahh]

Jython dynamically compiles its code to class files which are then loaded via a custom class loader and JITed by the VM.

I’m most interested in why tortoise “couldn’t be more happy with Jython”…

[sma]

[quote]In my perfect world, Java 1.5 would include it alongside autoboxing, generics, etc :).
[/quote]
Well, that would be Nice (Pun intended

My biggest gripe with all these variants or supersets of Java like AspectJ, Nice, Kiew, etc. is that you’ll get a better language but you loose the tool support. I consider the features of Eclipse or IDEA as a bare mimum of what is required to comfortable work on larger projects. Don’t tell me that refactoring needs no support because we all wrote Java 5 years ago without refactoring tools. Well, 10 years ago I already had a refactoring browser for Smalltalk and it was a big step back when I came to Java around 1997. The first tiny relief was VisualAge for Java which, written in Smalltalk by Smalltalk developers added back some comfort of the older language.

So I think, not only language featuers matter but also do we need at least as good tool support as we can have nowadays for Java.

[sma]

Funny you should say that…I have to make a decision very soon on what scripting language will be used in a browser-based-game whose heritage is that it used to be a PBM. At the moment, it looks like we’ll probably go with BeanShell for two reasons. One, *Python is useless because of concerns over editing source inside HTML pages (tab? Oh damn!). It could be done, but only by violating the language, and I’m not prepared to support a bastardised Python. Two, quick integration-time and simplicity are more important than perfection for this game.

While I can understand argument #2, why would you need to type a single tab? Just use spaces for indentation always and you should have no problems at all.

I see a problem with both languages, though. If you need other people than the core developers (some kind of game masters, directors, storytellers, wizards, etc.) to be able to edit code, I’d go for a language which makes it quite difficult to make errors or - to put it the other way - a language which makes it easy to write the right code. They will probably script existing objects/components/modules/functions. Those are prone of NullPointerExceptions and ClassCastExceptions then. Furthermore, if you create large amounts of scripting code, you need to be able to keep it up to date if for example a core function changes or if you just want to know whether the “coders” use the certain functions efficently or inefficently.

A static type system could help here because you can catch some errors at the upload = compile time. You could even provide a Java Applet for code editing whcih has the usual syntax highlighting and provides code assistents.

But there’re different kinds of static type systems. The one Java uses, is IMHO broken… near to useless. As long as you could get NullPointer and ClassCastExceptions, the static system adds no real value (you can never be sure whether the code will run without those exceptions) but only adds unwanted burdon, that is type casts.

You could get rid of mose clast cast exceptions with a more flexible type system that doesn’t assume class = type. You could also use a more powerful system with union types and type variables (generic types) that helps to express yourself.

You could get rid of nearly all of that dreaded NullPointerExceptions by removing “null” as a valid value for Objects of type X. If you need to express the fact, that a method will return either a String or null, then do it that way: “String|Null getString(int index)” You could add some sugar and write String# getString(int index) or something else.

But the result would be Java anymore and it would be a bit more difficult to interact with Java which now would have the less-powerful, weaker typesystem.

I nearly perfect language in this regard is Curl, but unfortunately that language isn’t freely available and of course, there’s no Java version. And because it’s quite powerful, it’s not that easy to recreate

[quote]Most of this game is written in Java, but that’s because most of it doesn’t need to evolve over time.
[/quote]
That’s a bold guess.

[quote]The ultra dynamic stuff needs to go in soon, e.g. quests, and there’s no way we’ll be doing that in Java.
[/quote]
Well, you could simply make each quest one or more classes, add a mechasim to serialize enough of the game and make the rest of the same not directly prefer to any of the quest classes and then use Java’s class loaders to dynamically reload stuff. Tomcat does the same for Servlets for example.

[quote]FYI for anyone wondering “should I use Java or [scripting language X]?” for a similar game, Java’s been a very good choice for most of it:
[/quote]
My first argument would be that you can deploy on any hardware that supports a JVM. Funny you not even mention that. My second argument would be, a lot of people know (and even more they think they know that language. My third argument would be that there are a lot of APIs and frameworks I could reuse.

Otherwise Java isn’t the fastest languages (I doubt that VisualWorks Smalltalk or Fun-O-Object’s Dylan implementation would be slower) and I wouldn’t give a damn on Java so called static type safty. Exception handling is something I’d expect from any modern language. Java can’t even restart a block - something Smalltalk, Ruby or Dylan have no problems with.

Creating a new custom language is probably no option for your game?

[blahblahblahh]

While I can understand argument #2, why would you need to type a single tab? Just use spaces for indentation always and you should have no problems at all.

Sorry - my inexperience with Python. I’ve spent only tiny amounts of time using it, and wrote that thinking that only tab indenting was allowed.

I see a problem with both languages, though. If you need other people than the core developers (some kind of game masters, directors, storytellers, wizards, etc.) to be able to edit code, I’d go for a language which makes it quite difficult to make errors or - to put it the other way - a language which makes it easy to write the right code.

This is an excellent point. But IMHO you miss some good solutions by leaping straight to conclusions on what causes those errors and encourages them.

I am the API God here - I define what types, classes, methods exist in the system.

[quote] They will probably script existing objects/components/modules/functions. Those are prone of NullPointerExceptions and ClassCastExceptions then.
[/quote]
I’d be very suprised to see many - if any - of either of those. In developing half a megabyte of source code there have only ever been fewer than 30 npe’s.

Its quite difficult to generate cce’s too, even in a typeless interpreter (although I’ve not yet tried on this project, so perhaps I’m missing something here?). None of the methods that will be scripted take any Object args other than “String”. Assuming the scripting lang can manage not to pass a valid int to an int arg as a String (I would hope they could all get that right :)) AFAICS cce’s will be very rare.

[quote] Furthermore, if you create large amounts of scripting code, you need to be able to keep it up to date if for example a core function changes or if you just want to know whether the “coders” use the certain functions efficently or inefficently.
[/quote]
This aspect is going to be quite an experiment (unofficially :)). The hierarchy has been built with multi-level development patterns in mind (i.e. low-level dev, mid-level, high-level - with appropriate unique usage patterns, amount of change over time, etc), and even though the core platform (3rd-party) has twice undergone major changes underneath us, the high-level code hasn’t needed to be changed at all (other than a few tiny changes to reflect no-longer-necessary workarounds etc).

But that, of course, is no guarantee that it’ll carry on working so well. We’ll just have to see…

But there’re different kinds of static type systems. The one Java uses, is IMHO broken… near to useless. As long as you could get NullPointer and ClassCastExceptions, the static system adds no real value

I’m not going to defend Java here, but IME whilst you are accustomed to thinking in java then runtime exceptions are few and far between.

OTOH, I mainly code network systems, servers, clustering etc. Traditionally highly defensive code. I love the clean integration of exceptions (and especially finally) into Java. Having all Throwable’s in a hierarchy, the duality of Exception vs Error, and being able to assign “default” handlers on a per-threadgroup basis is beautiful when you get used to it. I can make a complex app surprisingly robust with tiny amounts of code…although it does take a lot of time/experience to decide where precisely to handle problems. I’ve made the odd major mistake on that in the past and had egg on my face.

Well, you could simply make each quest one or more classes, add a mechasim to serialize enough of the game and make the rest of the same not directly prefer to any of the quest classes and then use Java’s class loaders to dynamically reload stuff. Tomcat does the same for Servlets for example.

I’ve been down that route before, and it can work very well. Right now I’m using a technology platform that really likes it when you only refer to things by type (not class); it helps enormously with modularisation, of course.

However, I’ve had problems before where I deployed it in situations where I then had to scale up the number of quests exponentially (this was in a research project; I had no idea in advance that I’d need to increase the dataset that much in order to meet my objectives). The 1.3 JVM didn’t like having a million objects; it coped, but guzzled RAM.

This game is expected eventually to have to handle over a million players…hence not using objects as args to methods, and wanting to avoid using many classes for one problem.

[quote]My first argument would be that you can deploy on any hardware that supports a JVM. Funny you not even mention that. My second argument would be, a lot of people know (and even more they think they know that language. My third argument would be that there are a lot of APIs and frameworks I could reuse.
[/quote]
…but the same is true of the major scripting langs. Probably more so. Certainly, running a game on a hosted server you can guarantee the existence of things like Perl, and easy installation of Python etc.

Otherwise Java isn’t the fastest languages

Oh, it’s partly because I’m one of those bigots who is still suspicious of many langs because I remember how slow they used to be ;D. However, I do have an excuse for this: I’ve been won over by Java’s NIO. Really, it’s such a relief to have just ONE asynch API that I’ve forgiven Sun for the dog’s dinner they’ve made of the docs (and much of the implementation)…and the fact that they included many of the newer asynch API features (and didn’t just copy completion ports, which are a nice idea, but perhaps too limiting?).

[quote] Exception handling is something I’d expect from any modern language.
[/quote]
Where’s the finally keyword in C++? I’m sure there’s a variant that supports it by now, but…

Creating a new custom language is probably no option for your game?

It is an option. I was (still am?) seriously tempted. I will ultimately end up with responsibility for training all the script-writers, so I can do what I want :). However, because of time constraints, I’d prefer to go with a pre-existing option, where I don’t have to write documentation and tutorials!

[blahblahblahh]

FYI…

I remember someone saying something about Beanshell not being able to extend classes, or something like that.

Just noticed the following from the current version release notes:

Previous limitations on the implementation of anonymous inner classes have also
been lifted, allowing BeanShell to extend arbitrary Java classes and implement
abstract base classes.

BeanShell 2.0 also brings with it two new language features:

JDK 1.5 style static class imports. You can import the static methods and
fields of a java Class into a BeanShell namespace. e.g.

static import java.lang.Math.*;
sqrt(4.0);

Instance object imports (mix-ins) with the importObject() command. You can
import the methods and fields of a Java object instance into a BeanShell
namespace.