Embedding Lua in Torque2D

For the past couple of weeks, I’ve been experimenting with integrating lua into Torque2D. If you’ve used Torque2D before, you may wonder “Why stick lua in if Torque2D already has TorqueScript?”. There are several reasons, one main one being many times I’ve felt TorqueScript’s lack of advanced language primitives (e.g. lists, tables, closures) to be an annoyance.

Adding simple keywords to TorqueScript is relatively simple since it uses a simple Yacc grammar coupled with an AST. However the implementation of the AST and the design of the stack makes it quite difficult to implement anything more elaborate. Even worse, TorqueScript has no unit tests, so you can easily break something if you are not careful.

So rather than having the burden of maintaining TorqueScript itself, I decided to try integrating another scripting language.

Its important to note that TorqueScript itself is tightly coupled with the object system in Torque, so its worth understanding how it functions before replacing it.

Brief overview of TorqueScript

TorqueScript is a fairly simple scripting language backed by a byte-code interpreter. Internally all variables are strings, though they are simultaneously casted to float and integer types to speed up any math operations. All variable access goes through either a local or global hash table (apart from the name of the variable itself, nothing is cache’d).

Local variables are prefixed using the % sigil, and global variables are prefixed using $. Some variables are registered by the engine itself, in which case they directly reflect whatever variable they point to.

TorqueScript’s object system is rather ingenious. Objects are referenced by either a numeric id or a name with optional quotation marks. You can also use more elaborate search strings, e.g.


1234
"MyObject"
"RootGroup/MyObject"

May all refer to the same object.

As far as the scripting engine is concerned, objects have two things: a list of properties, and a namespace. Through the namespace functions are accessed. The class hierarchy is mirrored through the use of linked namespaces.

When accessing properties, it searches for the property in the class property list first, then looks in the objects property list. When accessing functions, it checks the classes current namespace for the function. If the function doesn’t exist in that namespace, it checks the parent namespace and so on  – though internally this check is cache’d using a hash table in the first namespace which contains the complete set of functions.

Example Namespace Linkage (1)

When you create an object you can also give it a name, in which case it will link in an extra namespace at the top of the chain allowing you to do cool things such as this:


new SimObject(FudgeCollector);

function FudgeCollector::collectFudge()
{
   echo("Collected fudge");
}

FudgeCollector.collectFudge(); // prints "Collected fudge

Namespaces in TorqueScript don’t necessarily have to be linked to a class. For instance you can create a new namespace at any time by defining a function.


function MyNewNamespace::doThis()
{
   return 123;
}

MyNewNamespace::doThis(); // No problem

Internally all objects are derived from a ConsoleObject class which handles fields, though most of the action happens in the SimObject class which handles assigning names and identifiers to objects, and well as setting up namespaces. There is also a SimSet class which adds nesting support.

Safe to say TorqueScript is a fairly loose and flexible language.

On to lua…

“So why lua?” you may ask. Several reasons. Firstly out of all the languages I tried, lua seemed to be the most flexible. This meant that it was reasonably easy to implement something which resembled how TorqueScript functions. There is no real class system, so you are free to invent your own using linked metatables. Variable access can easily be overridden so you can expose globals and object properties pretty easily.

It’s also ridiculously easy to bind lua code in torque using the current TorqueScript binding system. Instead of having to adding a lua thunk function for every function and method, it’s possible to use a single thunk function and use “upvalues” to point to the correct C function. Although ultimately it would be a better idea to use a specialised thunk for each exposed function, the fact that it isn’t strictly required reduces the complexity of integration  substantially while ensuring existing TorqueScript code can still function without the need of a rewrite.

Comparing lua to other embeddable scripting languages with class systems (e.g. ruby, python), they tend to be a bit stricter when it comes to object classes, making things such as the namespace name trick harder (if not, impossible) to implement correctly. i.e. you can’t just create an object on the fly and bind new methods to it, you need to make a proper class or make the functions references to delegates.

Lua does have a few of its own oddities – for example, arrays start at 1, and there is no switch statement. But these are a reasonable trade-off for flexibility.

On to the binding…

To be honest, it took me a while to get the right implementation which reflects the way things already work in TorqueScript. Since I wanted to keep the existing binding system, I had to write my own bindings from scratch using some of the existing C++ binding libraries as references to how things should work on a basic level.

For my implementation I ended up using userdata objects bound to class metatables. The userdata objects point directly to the instance of an object. Property access goes through one of two thunk methods (__index and __newindex), while functions are binded through one of 11 thunk methods depending on the return type and whether or not the function is meant to be used with an object. Functions are assigned to the relevant table in lua (e.g. SimObject methods go in the SimObject table, global functions go in Torque). These tables are linked together to reflect the linkage of each namespace in Torque.

Lua Namespace (1)

When looking up properties, the __index method first checks the relevant top namespace table in lua. If it doesn’t exist there, it checks the parent table, and so on. If nothing is found in the end, it assumes the key is meant to refer to a property and acts the same way as in TorqueScript when you get a property.

When creating objects, the equivalent lua reference is stored in a global table called InstanceTable. e.g. “InstanceTable[1234]“.

So for example the following lua script…


myObject = SimObject()
myObject.customValue = 'candy'

function SimObject:printInfo()
   Torque.echo(self:getId())
   Torque.echo(self.customValue)
end

myObject:printInfo()

Will print the id of the object “myObject” and the word “candy” in the console.

In the engine code, everything is abstracted into a LuaScriptEngine class which handles binding classes, registering objects and functions, and also executing code. There’s also an equivalent implementation of the “Con::executef” function for calling back into script. Unlike the TorqueScript executef, this implementation uses a list of ScriptStackValueRef objects which transparently handles pushing pushing values onto the lua stack with the desired type.

Unfortunately one thing I haven’t implemented as of yet is nested objects, where you can create an object within an object and have the script engine automatically add it to that objects group for you. However given that Torque2D uses TAML to serialise nested objects this feature can be safely ignored.

Conclusion

Screen Shot 2014-02-03 at 20.17.33

Looks like lua

So was it worth embedding lua into Torque2D? I’d say yes. At the very least I now know how deep the TorqueScript rabbit hole goes, making it easier to incorporate any of the other alternative embeddable scripting languages should they become a better fit for my needs.

Torque2D Emscripten port sourcecode released

Last month I unveiled a port of Torque2D which runs in web browsers via emscripten. It was developed largely over a period of a couple or weeks with the help of a few emscripten developers along the way.

After the unveil it generated some interest, but not really enough for me to make a product or anything commercial out of it especially given the current trend for game development engine tech: release everything for free.

Considering this, I’ve decided to release the code for the emscripten port under the same lovely MIT license as Torque2D itself. I hope someone can find it useful, perhaps to help promote their Torque2D game. Or who knows, maybe they can release a web-based browser game for it! The way web browsers are heading, I guess anything can happen!

Scripting Language Shootout

Picture this scenario. You are developing a game engine, but get to an all-important question: “which scripting language do I use?”. The problem is, there are tons of scripting languages out there. Which one do you choose?

Unfortunately, picking a scripting language is not always easy. A lot of the time its a trade-off between performance, support, and something you like. There’s not much sense in picking the fastest scripting language in the world if you hate it after all.

For me, I don’t particularly care about the speed of a language (this has been explored before), so long as it’s reasonably fast. After all, if I’m going to implement something CPU intensive, I’m probably going to just end up writing it in C. I’m also open to anything, so long as it meets the following requirements:

  • Reasonably easy to embed
  • Uses an encapsulated state instead of globals
  • Has a capable binding API
  • Has some sort of OOP system with objects bound with functions
  • Is used by a community of people

Since I had time to spare, I decided to evaluate a few popular scripting language implementations which have been described as being embeddable and fit the above criteria. Basically I wanted to:

  1. Bind an object to the API
  2. Pass an existing instance of an object to the API
  3. Create an object in the scripting language
  4. Call methods and get/set properties on the object
  5. Handle errors sanely (i.e. no stack corruption)

In addition I didn’t want to rely on any third-party wrappers – instead I went straight to the C API, since otherwise I would have been evaluating the scripting engine + the “helpful” binding library, rather than the scripting engine itself.

I ended up making a test application which binded a test object to each target language and ran a sample script in order to get a feel for how everything worked. The code is on github, so feel free to check it out if you are wondering what its like to bind a C++ object to lua, mruby, squirrel and angelscript.

I wrote a few notes in the code, but I felt the need to express my thoughts on each language based on my experience in working with both the API and language myself. So behold! My evaluation of each language is as follows…

Lua(http://www.lua.org)

Lua is a powerful, fast, lightweight, embeddable scripting language – or so it is claimed. It seems to be the number 1 choice for game developers, as this wikipedia article shows.

Scratching the surface, it appears to be a simple functional programming language. However it features some very powerful constructs centred around the concept of a “metatable” which amongst other things can be used to create class-based object systems. The API is well documented, and its fairly easy to find information on how to do just about anything in Lua since its commonly embedded in apps and games.

An often-heard complaint about lua is how it indexes arrays from 1, though in practice I don’t really find this to be much of an annoyance especially considering luas support for iterators. Another issue is how it isn’t very C-like in general, but IMO this isn’t too big of an issue since the syntax is fairly trivial to learn and get used to anyway.

The binding API in lua is a bit confusing as it uses the stack to set fields on objects, which almost feels like reading some assembler code. Once you realize how the stack works though, it’s not too bad.

Binding objects themselves is an open-ended problem, with many solutions. At a bare minimum you need to create a metatable object for the class, and for each instance of the class you need a userdata object bound to the metatable. You can just use a userdata for object instances and implement “__index” and to return methods, but things start to get tricky when you need to call functions as if you just return a C function from “__index” lua won’t provide any reference to the userdata, so you need to pass in whats known as an “upvalue” when returning the function.

UPDATE: turns out it does provide a reference to the userdata, you just need to use the “:” instead of “.” when calling the function. Basic Lua 101.

One gotcha with the C API to look out for is when calling methods within methods. If the method you are calling returns an error, lua will pop the C stack all the way down to your first lua_call. This will ignore destructors for any stack variables you still have in scope in-between. To mitigate this, make sure to use lua_pcall instead.

For derived classes you can simply make a new metatable and set its “__metatable” to the parent class metatable, ad infinitum.

Apart from that binding objects to lua is reasonably simple, and if you are having problems there are a ton of binding libraries out there to help. A fairly strong contender.

Squirrel(http://squirrel-lang.org/)

Squirrel is a high level imperative, object-oriented programming language, designed to be a light-weight scripting language that fits in the size, memory bandwidth, and real-time requirements of applications like video games – or so the squirrel website claims.

In reality, I like to think of it as the result of what happens when someone takes a lot of the bad things about lua and fixes them to create a rather neat C-like language.

My only big problem with squirrel from a language perspective is its odd use of the “<-” operator to set object slots, though this can be “fixed” by simply changing the bytecode used by the “=” operator.

Similar to lua you use a stack to bind functions and classes which can be confusing. Thankfully though as with lua there are comprehensive API docs, but with one fatal flaw: I couldn’t find any good example for how to create a class bound to a C++ object, so once again I had to guess. I could excuse this a bit in lua since it’s more ambiguous, but in squirrel classes are a first level construct, making it strange how it isn’t really better explained in the documentation.

In general while I found the API documentation to be comprehensive, I didn’t feel it explained how to do things very well, and combined with the relative lack of resources online it made things a little harder to figure out.

If you are fed up with luas syntax and lack of certain basic constructs, be sure to give squirrel a try.

mruby(https://github.com/mruby/mruby)

mruby is a light-weight, embeddable implementation of the Ruby programming language which has been in development for the past couple of years, so its fairly new.

Ruby itself is a very capable dynamic, object-oriented language which to me at least feels like a cross between perl and python. Ruby also has a proper inbuilt class system which IMO sets it on a higher level than lua. Not only that, it even supports composition through the use of modules.

Unfortunately the binding api is a little under-documented. In fact at time of writing I couldn’t find any documentation other than blog posts. In order to write a binding I had to essentially guess what you had to do using a combination of code on github and deduction.

However upon reflection binding an actual object is rather simple, with some rather descriptive function names to define both the class and its methods. For object instances, you use userdata instances bound to a data type.

Similar to lua there is also a gotcha when calling methods within methods. If the method you are calling raises an exception, mruby will pop the C stack all the way down to your first mrb_funcall. This will ignore destructors for any stack variables you still have in scope in-between. To mitigate this you need to save the “jmp” variable before the call and restore it afterwards. If you don’t want the exception to propagate down the stack you also need to clear the “exc” variable. If mruby had mrb_funcall_safe, this hack could have been avoided.

Compared to lua & squirrel, ruby is in general a more extensive language. But mruby itself still feels like it’s not quite there yet, so if you are considering implementing ruby in your app you might want to consider embedding the reference implementation instead.

Angelscript(http://www.angelcode.com/angelscript/)

Angelscript is a real curiosity. In essence it answers the question: “What would happen if you made a scripting language out of C++?”.

What results is a strongly typed, inflexible language. When compiling, everything needs to be defined or declared up-front. There are no fancy metaprogramming constructs, nor any late binding of functions. From what I gathered, if you make your whole game in a set of AngelScript files, you need to compile everything together in a single module. It makes me wonder, why not just use a reloadable DLL written in C++?

Unlike C++ Angelscript prefers that you define all class and namespace methods within the class and namespace blocks, making it feel more like C#. This is a fairly minor issue, but if you start thinking “maybe I’d like it better if I just wrote everything in C++”, it does not bode well for the scripting language.

However out of all the scripting languages, I’d have to say AngelScript was by far the easiest to embed. You simply define your function prototype and BAM! It figures out how to call it without having to write your own binding function (using its own FFI system). You can also directly expose object fields too which is a nice touch.

If you want something more direct, Angelscript is a safe bet. If you want something more freeform, look towards the other scripting languages.

Conclusion

So which scripting language do you use? Ultimately there is no perfect solution, you just have to go with the one you feel is the best. Failing that, you can always make your own scripting language, which is not at all an unusual thing to do when making a game engine!

 

 

Introducing libDTShape

When developing 3d games, the allure of using a pre-built game engine is all too great. The trouble is nowadays you really need a good content pipeline in order to take 3d objects from your modelling tool and render them in your game, which in the past has usually required someone on the development team to write an elaborate tool-specific exporter script or plugin.

Fortunately nowadays with solutions such as collada you can now pretty much skip the “write exporter” stage and move the logic to generate runtime model data into the engine itself. However you still need something to handle animation and skinning. There are a few solutions out there, but they either contain too much baggage or the licensing isn’t so great.

So I decided to try making something myself. My requirements were as follows:

  • Written in standard C++
  • Engine-Agnostic
  • Renderer-Agnostic
  • Cross-Platform
  • Proven codebase

Most importantly I wanted to base my solution off of something I knew already worked, and I knew just the thing: the “dts” code from Torque3D, which has been in use in some form or another for over a decade.

DTS itself is quite a featured format, being suitable for rendering level geometry with basic culling and fully rigged and skinned character models. In Torque3D there is even a whole content pipeline so you can go straight from collada dae to “compiled” dts shape.

You might be thinking now “Wait a minute, this DTS stuff must be tightly integrated into Torque3D. Won’t I have to include 99% of Torque3D to use this?”

This is where the beauty of libDTShape comes in. libDTShape is an extremely stripped down version of the DTS code from Torque3D. It includes only the essential code which is needed to both load and create a DTS shape, while leaving you to implement the rendering. It lives in its own namespace, so it can work independently of your own code. It is also configurable, allowing you to strip down the code even further (e.g. to remove collada support).

For those who want to just quickly load a shape and see what you can do, don’t worry! libDTShape also includes a basic example OpenGL renderer.

libdtshape

 

Admittedly as it currently stands libDTShape is a bit of an experiment, but still I hope this can be useful to others.

libDTShape can be found here: https://github.com/jamesu/libdtshape

 

Torque2D… in a web browser?!

For anyone following the indie game development tool market, there’s one thing that just about every popular commercial tool has: the ability to export your game to the web. Being able to run your code in a web browser is an extremely powerful tool, not only from a platform standpoint but also because it eliminates the requirement of having to install anything.

The tools for doing this in the past sadly either required the end-user to install something (if you made your own plugin), or conforming to a strange runtime (flash, unity, java). Torque itself at one point went with the “make your own plugin” approach, which to say the least was a disaster since every developer had to go through the hurdle of making a plugin for each major system and ultimately the end-user had to install something. Unless your game is the best thing since Minecraft, this approach will never work.

Thankfully now there are several more promising solutions on the horizon, one of which is Emscripten. Emscripten is a compiler that takes standard C/C++ code and compiles it to Javascript suitable for running in any modern web browser. In essence as long as you have a modern 3d graphics card and HTML5-capable browser, you should be able to run just about any game code on all major platforms!

Getting to the point, over the past few weeks I’ve been working on a port of the Torque2D game engine to Emscripten. After a bit of help from the Emscripten developers, nearly everything works (excluding networking).

Torque2D Emscripten

On a time scale, the basic port took a weekend while the bug fixing was spread over the next few weeks. Porting across code to Emscripten was relatively easy since it exposes a pretty standard unix API with SDL, OpenGL, OpenAL, and a few other useful APIs. However the difficulty was multiplied somewhat when I discovered some rather nasty memory corruption bugs.

Firstly there was a problem with the dynamic_cast handler, in which a cast would fail for no apparent reason. It turns out a key map in the input code was a bit too small, causing an overflow which corrupted some memory related to handling the cast. There was also a problem in the scripting engine in which script strings would randomly be corrupted which I ended up hacking around.

Sadly Emscriptens debugging facilities were not up to the task in this case. While there were several facilities to debug memory corruption, none of them picked up on these issues. Unfortunately this meant  I had to go through every major bit of code to eliminate what was causing the problem, which took a considerable amount of time.

Another thing to note is Emscriptens OpenAL implementation was missing a few crucial functions, though these were fairly trivial to implement.

Overall I was quite impressed with Emscripten, though it would be nice if there were more debugging facilities or even some form of GDB frontend so I could more easily step through code.

For your enjoyment, A build of the port can be found here: http://stuff.cuppadev.co.uk/torque2d-web/

 

 

Current Status

For those who stumble across this blog, you might wonder why I haven’t updated this blog for over a year. Truth be told I’ve been busy working on a few projects for Mode7 Games such as  Frozen Endzone, which took up a considerable amount of time.

Apart from the problem of not being able to find time to update, I also had a problem of not being to update at all. A while ago I migrated the entire site to Jekyll, a static site generator. This worked fine for a while, but I quickly grew tired of the long delay when regenerating the site when typing up new posts.

I also felt a bit of disconnect needing to type up each post in a text editor and having to guess how the post formatting would turn out. In short it felt like a big chore to post anything. And for me the ability to post things quickly and get instant feedback is a much needed attribute of a blogging system. Thus I have now migrated back to WordPress.

I have a few projects I would like to release before the end of the year, so expect me to write about them here. I’m also going to write about what actually happened this year, including what went wrong during the rather disastrous deployment of the Frozen Synapse server rewrite.

Stay Tuned.

Optimizing TorqueScript… Again

For the past few weeks one of my weekend side-projects was to improve TorqueScript, the scripting language behind the Torque3D game engine.

This time I decided to tackle TorqueScript’s type conversion system. This basically handles converting strings such as “0.1” into native values such as 0.1f, and vice versa. Since everything must be converted to and from strings all the time when manipulating object fields and object variables, it’s a great bottleneck for performance if scripting is used too heavily.

A Little Background on the Type System…

In TorqueScript, values can be one of four types:

  • Strings
  • Unsigned Integers
  • Floats
  • Custom Types

The first three are self-explanatory. The fourth is more involved as it is used to expose engine variables and object fields as strings. Getters are implemented using the ConsoleGetType macro which implements a function returning a string based on the value, setters are implemented with the ConsoleSetType macro which implements a function which parses a string and sets the value.

All access to registered engine variables and objects goes through this type system, so any improvements to it have a good potential to improve overall performance. For instance point values in TorqueScript are always manipulated as strings, so any complex vector math incurs the overhead of string conversion. For this reason I also decided to implement value-based arrays.

What went wrong

One big problem with altering something as monolithic as TorqueScript is little changes can cause big problems since the whole implementation is essentially an inter-connected mess. Changed the syntax? Be prepared to debug the AST. Changed the AST? Be prepared to debug the bytecode. Changed the bytecode? Be prepared to debug the interpreter.

For instance, when changing parts of the AST I frequently found that it wouldn’t generate the correct opcodes. The issue was that the AST nodes had been designed assuming the other statement nodes behaved in a specific way, so when I added new variable types to support arrays it completely failed.

In another case simple changes to the interpreter and the AST caused the float stack to become unbalanced, which required painfully tedious step-by-step debugging in order to resolve the issue.

Problems from the ground up

TorqueScript suffers from a key design issue: stack values are split into three pools: strings, integers and floats. The only generic value type is a string e.g. when calling functions, the compiler has to assume the return value is a string.

There is some code present in the interpreter which skips the conversion if the next opcodes indicate the value will be converted to an integer or float, but this is more of an elaborate hack around a poorly designed system.

Since there is no generic stack value, operator overloading is impossible (thus the need for specific string concatenation operators). Opcodes relating to field access need to be duplicated in order to handle different types. Object references need to be resolved every time a field is accessed or when an object function is invoked.

TorqueScript also lacks tests to determine if it functions correctly or not, which ultimately means everything needs to be manually tested in order to verify it still works correctly… unless you have the time and resources to create a complete testing suite.

To Conclude

While this project has been interesting, ultimately I’ve decided to shelve it for the time being.

It turns out fixing a scripting language is harder than it first looks. While I managed to implement arrays and improve the way certain types are set, fixing all the bugs caused by the changes turned out to be an even bigger hurdle. In the end, I was stuck trying to resolve some tricky killer bugs in the editor scripts.

In light of my recent experience, I have the uttermost respect for anyone who maintains a scripting language.

For anyone interested, this code is available in my Torque3D fork.

Optimizing TorqueScript

For the past few weeks one of my side-projects has been to improve TorqueScript, the scripting language behind the Torque3D game engine.

TorqueScript itself has a long history, going all the way back to Tribes 1 in the form of TribesScript. After years of maturing, one might thing this is an optimized world class game scripting language. Sadly this is not the case.

I came to this conclusion after creating a basic series of tests which benchmark a simple case of adding numbers together 1000000 times in TorqueScript.

For reference, the initial results I got for these tests were:

# Runtimes are in milliseconds
Test,1,2,3,4,5,AVG
scriptTest1,6880,6464,6496,6464,6464,6553.6
scriptTest2,640,672,640,640,640,646.4
scriptTest3,4960,4832,4832,4832,4832,4857.6
scriptTest4,512,512,512,512,512,512
scriptTest5,3104,3136,3104,3136,3136,3123.2
scriptTest6,256,256,288,256,256,262.4

Compare that to the sort of times you get from Lua ( ignoring tests 4 and 5 which tested bindings ):

TEST,1,2,3,4,5,AVG
scriptTest1,177.313000,173.866000,173.777000,173.635000,172.777000,174.273600
scriptTest2,365.673000,366.093000,366.950000,367.679000,373.810000,368.041000
scriptTest3,495.440000,493.399000,490.451000,492.778000,491.064000,492.626400
scriptTest6,151.421000,150.907000,150.767000,149.743000,151.732000,150.914000

That is a shocking ~37x difference in speed.

The problems

In TorqueScript, variables can be one of three things: an integer, a floating point number, or a string. Integers are commonly used to perform bitwise operations, while floating point numbers are used for math. You can also have binded variables which work as complex native types, though they are converted to and from strings when manipulated from script.

Most of the time, a variable ends up being a string since that is the only default assumption which can be made. For instance, if we assign a local variable like this:

%variable = %otherVariable;

It will generate a byte-code sequence like this:

OP_SETCURVAR "otherVariable"
OP_LOADVAR_STR
OP_SETCURVAR_CREATE "variable"
OP_SAVEVAR_STR

In essence this will convert “otherVariable” to a string, regardless of its original type. This is not too much of a problem, as the integer and float values are stored along with the string value. However if we do something like this:

%variable = %variable + 1;

It’s quite clear to the compiler that %variable should be a floating point number so it will generate a byte-code sequence something like this:

OP_LOADIMMED_FLT 1
OP_SETCURVAR "variable"
OP_LOADVAR_FLT
OP_ADD
OP_SAVEVAR_FLT

A problem comes with the OP_SAVEVAR_FLT opcode. This will destroy the string value of the variable and change its value to a floating point number. If we do something like this again:

%variable = %otherVariable;

It will convert the variable back to a string (allocating memory in the process), so in essence one ends up with a variable being dragged between two states: being a string and being a number.

“But wait a minute,” you may think. “most of the time it’s clear that a variable should be a number.” This is true until we get to function calls. In TorqueScript all function parameters are passed as strings (even binded functions), so if you do this:

doThis(%currentVariable, 10, 20, "MyName");

For each parameter it will copy its string value onto the “String Stack”. When the function is finally executed, it will copy the variables from the string stack and assign them to local variables, or in the case of a binded function it will just supply you with a series of strings pointing to the stack. Upon returning, the return value will usually be converted to a string.

So not only do we have to contend with numeric variables being strings in the function, we also have to keep in mind that any string passed in will likely be duplicated twice too.

For any scripting interpreter, allocating and deallocating memory unnecessarily can be very costly. For Torque3D, this problem is compounded by the fact that it by default uses the system memory allocator (malloc/free) which can differ greatly across platforms. In addition there is no Garbage Collection, so everything is deallocated as soon as it is disposed of.

The solution

So what’s the solution to this mess? For starters, optimizing variable assignment helps. Creating “load variable” and “save variable” opcodes so “%variable = %otherVariable” turns into this:

OP_SETCURVAR "otherVariable"
OP_LOADVAR_VAR
OP_SETCURVAR_CREATE "variable"
OP_SAVEVAR_VAR

Ends up making a measurable difference in performance when assigning lots of numeric variables.

Next, function calls. Instead of converting every parameter and return value to a string, wrapping them in a “Console value” class to eliminate unnecessary type conversions which transparently works like this:

ConsoleValueRef argv = {"arg1", 2, 3};
argv[0] // "argv1"
argv[1] // 2
argv[2] // 3

Brings the biggest speed improvement of them all. With these two basic improvements, I managed to improve the runtime speed by 8x in the best case:

Test,1,2,3,4,5,AVG
scriptTest1,784,752,768,736,752,758.4
scriptTest2,976,944,960,944,960,956.8
scriptTest3,1200,1216,1200,1200,1200,1203.2
scriptTest4,448,432,464,432,448,444.8
scriptTest5,944,928,960,928,944,940.8
scriptTest6,288,320,304,320,304,307.2
FIN

This is sadly where I decided to stop. The next major improvement, to improve object field access (e.g. “object.foo”), would have required a complete overhaul of the type system. For now, I’m just happy I got a nice speed improvement.

It’s somewhat surprising that despite ~14 years since inception, nobody else seems to have taken the step of making this basic beneficial improvement to TorqueScript.

For reference, my code with the optimizations is available here

Torque3D goes Open Source

Today GarageGames announced they were opening the code to their flagship 3d game engine, Torque 3D.

Torque3D Logo

Torque was originally released 10 years ago under the name V12 for the low price of $100. Since then it has spawned numerous games as well as spin-off products such as Torque 2D. It pretty much kick-started my software development career, my first major project being an export script to convert Blender scenes to Torque DTS models. Over the years i’ve had a good stream of work relating to Torque, e.g. working with Mode 7 Games on Frozen Synapse.

Open sourcing a key product is a bold move, and it will be interesting to see how well it will pay off. Nowadays with UDK, Unity, and a deluge of other game engines competing for developers, the market for paid-for royalty-based game engines has been looking a bit crowded.

World Builder Example

Personally I think Torque3D stands a good chance compared to other Open Source solutions. For one it has a fully featured world builder, content pipeline and GUI system. It also features some fairly impressive networking code. I can’t think of any MIT licensed open source product which provides all of these things in a single package all ready to use out of the box.

Torque3D does however have a few weaknesses. In a world full of dynamic languages, TorqueScript is looking a bit dated and has a few problems with its architecture. Multiplatform support is there but the OpenGL renderer is poor. Also while the rendering in Torque3D is good, it could be much faster.

Despite these issues I still think Torque3D is a more than capable game engine. I look forward to seeing what the future holds.

Revisiting Android Development

A long while back, I wrote an article which amounted to ”the android sdk sucks.” Since then i’ve had quite a few comments about it, so I’ve devicded to re-evaluate Android Development.

The platform

Since 2009, android had gone through 3 major releases. There are literally hundreds of devices you can target, stuck running everything from 1.6 to 4.1.

In a way this is similar to PC development. You have literally no idea what sort of hardware or OS people are going to run your software on.

Thankfully nowadays Android has a fairly sophisticated set of options for specifying minimum hardware requirements in the application manifest. Starting with version 3 there are also vast improvements in support for different screen sizes and resolutions.

Unless you are doing something incredibly weird or want to support absolutely every device, hardware diversity is the least of your problems with the current SDK.

The SDK

The Android SDK is heavily influenced by what I like to refer to as the “Java mindset”. When developing an app expect to be dealing with lots of XML files, complex Java abstractions, and strange tools.

It’s also quite large (though not as large as the iOS SDK) – I easily spent an hour downloading both the base SDK and the extra packages for eclipse development.

Eclipse is still the recommended IDE for development, though you can still run all of the tools from a terminal if you find it completely intolerable.

IDE

The only major improvement I noticed with the current eclipse plugin is that the GUI builder has greatly improved. Honestly, I now feel as if I could implement a proper GUI with it.

Emulator

The Android Emulator, your only way of accurately testing an app without buying a real device, is still sadly quite slow. After turning on GPU emulation (which is oddly disabled by default), performance improves slightly though it still feels quite laggy. If I was going to do Android development professionally I would definitely get a real device.

Documentation has improved by miles. There are a lot more useful examples of how to do things, and the documentation for classes has greatly improved. I can actually follow it and not screw up.

Last time I mentioned the documentation for writing OpenGL Applications was a bit lacklusture. I can safely say this is no longer the case – there are now useful examples, both in the examples and the actual class documentation. Not to mention, OpenGL support on devices has definitely improved miles.

If your codebase relies heavily on C, Android provides a Native Development Kit (NDK). I haven’t looked much into this, as the process to use it seems fairly overly complicated and doesn’t appear to be integrated well with eclipse.

The Android SDK is of course not the only way to develop for Android. There are also solutions such as Mono for Android and Unity. I haven’t used any of these, but I have heard good things about them.

To Conclude

Both the Android SDK and Ecosystem have definitely improved by miles since the last time I looked at them. In a way, Android has matured.

While the SDK is still a bit rough around the edges, I think if you get used to it, you should be able to make some good apps with it.

Maybe i’ll be developing some Android apps in the future…