Progress

And so the new speedy parser is done. Well, done enough that I can get useful performance data out of it. Technically, it needs integrating into the translation engine, but it works pretty well. Also LMU files, but that should be less than 20 minutes work once everything else is ready.

So, the summary of improvements: Much faster (10 times, at least), and now picks out "Change Hero Class Name" (thanks shadepariah for pointing this out and providing a sample of it). Also, (untested) should pick out if a face is being displayed on the dialog - this is useful for translators as a face decreases the amount of space available for text. Finally, the script parser should be much more robust than it was before.

Also note: as this parser uses a different internal representation, there won't be a release until a new serialiser has been written.

The rest of this post is a programming type post, for people interested in the intricacies of programming high performance Python. OK, a lot of this will also apply to many other languages, especially interpreted ones, but mainly Python. If you're interested, look after the break.



1) Lambdas are bad

For functional programmers, Lambdas are really useful tools, but in Python they are bad news. Why? They're slow, and they all show up as "lambda" in a profiling. That means it's impossible to tell where your bottlenecks actually are, and hence you can't improve performance easily.

If you need something like lambdas, I'd recommend a function factory now. Python lets you do them easily (try defining a function inside a function; every time the function is called, you'll get a new function. Now put variables from the parent functions local scope into the child function, have the parent return the child function, and you have a function factory).

2) Use language features to do as much heavy lifting as possible

Old RPGMaker Trans used a very verbose representation for constructing RPG Ints, by converting characters to bitstrings to ints. Now, whilst I did have a good reason for this in reverse engineering it (specifically, that I needed to see what the RPG Ints looked like), this hammered performance badly. The new version uses Pythons struct library for unpacking the file into ints, then bitwise math for the rest. Much better.

Another things to point out: don't use a custom object when a language native one will do. Initialising objects is much more expensive if the interpreter has to think about it, so stick to language types (like lists, dicts etc) as much as you can.

3) Data copying is bloody expensive

Old RPGMaker Trans copied data around in memory like nobodies business. Not a good idea, as it turned out. Compounded by the fact that Pythons string construction is not terribly efficient

4) It's cheaper to do one big operation than hundreds of small operations

In an interpreted language, function calls have high overheads. In fact, any operation has a high overhead. Simple message: if you can at all help it, batch up operations and do them all in one go.