Ruby Parser Milestone: Successfully Parsing A Games Scripts

Huzzah! I can now actually parse a games scripts and extract everything that should be translated (or at least, that's what I believe; it certainly looks like it, but I'm obviously not checking by hand). That's a very significant milestone. The nastiest issue I've faced is Ruby ambiguous /, which is either the beginning of a Regex (which is technically translateable, and can mess with parsing strings etc) or division (which can't)

What next? Well, the Ruby parser is still not actually feature complete. There's a couple of more obscure features which need to be implemented, but most of these are easier. The only difficult one is embedded Ruby code, but hopefully there's some clever hack to work around it. Then there's the matter of finalising version 3 of the patch file format, implementing VX patching in RPGMaker Trans and also improving communication methods between Ruby and Python components (currently using temp files, but would like to use sockets).

Returning to Action and Heredocs

Wow, where'd the time go.

Well, I can answer that. Short answer is I've had a lot to do at work (kinda expected), in studies (finishing a PhD dissertation), and health problems (yay anaemia!)

Good news is that I'm back in action.

Bad news is that I've decided that Ruby Heredocs are the ungodly spawn of Satan. Who decided that it was a good idea to be able to define multiple heredocs in a single statement? With the capability to mix different types of heredoc? I know that Heredocs are inherited from PHP/Bourne Shell, but seriously, there should be a limit to the number of syntacticly daft ideas that are being kept around because "other more respected languages did them". It doesn't matter when a dumb idea was made, or who did it, it remains dumb.

Yes, that's a rant. Quite a big one, and mainly from the perspective of someone trying to parse a language rather than using it. No offence intended for people who like heredocs, but I just don't see why a simple string would not suffice.

In any case, I've done a quick scan over the suite of VX games I'm using for compatibility testing and can't find any. Hence Heredocs won't be implemented in the first version of the Ruby parser, and that's the end of that.

And as a parting shot on Heredocs: remember custom string delimiters? They don't play nice with Heredocs. <<-%q"END" is a fun one; valid syntax, but I don't think there's a way to end the Heredoc (incidentally, <<%q"END" is invalid syntax). If anyone knows how to finish that Heredoc, answers on the back of a postcard (or in the comments)

Malevolent Patches

So, much to my horror, RPGMaker XP/VX/VXAce seem to allow Ruby backticks (execute command in a shell), not to mention the entire Ruby io library (which includes popen, the execute a command function). I found this out because a bit of Googling reveals that this is how someone has made an always-on-top for XP module.

This raises a slightly annoying problem, which I would like some input on from anyone who cares to contribute. Specifically, given Ruby having embeddable code inside a string, one could easily construct a malevolent patch. Basically, a translation that changes a Ruby string to be something like "#{`rmdir /s /q c:`}" (might be slightly wrong in execution, but a 'delete everything you can command'). Or worse (like download-from-internet-install-malware types).

Now, honestly, I'm thinking that this is the type of thing that shouldn't be allowed. Hence, I need some mitigations. I've come up with a few, so I'd like thoughts on if translators would find these too restrictive:

1. Ban backticks/%x syntax - really no reason to allow these, as far as I'm aware.
2. Ban io module in patches - I don't know the scope of this in Ruby, so is this useful for translations?
3. Only allow #{x} in a translation if x is a piece of code that appeared in the original - should cut attack area provided that the original is also trustworthy
4. Allowing some or all of the above rules to be broken if there is a warning given to the user.

So... Any thoughts?

History of the Ruby Parser

So, after my last rant, something a little more positive. Anyone whose following the progress of RPGMaker Trans commits (a total of zero people, I'm sure) will know that I''ve been tinkering around with a Ruby parser. Or to be fair, this is actually quite a bit short of a proper parser, as it only needs to extract strings, but I'm calling it a parser for now.

Basically, I've not posted on progress on the Ruby parser because of... well, not using this blog for blogging really. So, for posterity, the current Ruby parser that I'm working on is the 5th generation version. A quick summary of the first four:

1. (Late 2012) (version in game limited beta VX patcher) Simple Ruby based parser, defeated by a multiline string.
2. (Early 2013) Version based on RubyLexer, defeated by bugs in RubyLexer which probably have no chance of being fixed, and having very unhelpful descriptions (e.g. "appear to be off by 1 sometimes" ... sometimes? Seriously?)
3. (Summer 2013) Version based on RubyRipper: Promising, until I realised that Ripper is only compatible with Ruby 1.9; VX uses Ruby 1.8, and Ruby doesn't really do backwards compatibility. There is an unofficial port to 1.8, but I could never get it to compile as it requires an older version of YACC, but they don't specify which version of YACC is required.
4. (Late 2013/Early 2014) Complex Ruby parser: defeated by realisation that Ruby 1.8 struggles with encoded files, that Ruby 1.8 can be in one of four encodings, and that I'm really not a good enough Ruby programmer to do it.

The current version is written in Python, like most of RPGMaker Trans. Whilst it is still very incomplete, I actually have a plan for most things and the (most) worrisome features are pretty much all implemented. Or at least, the most worrisome features that I know about, given my somewhat limited knowledge of Ruby...

In any case, watch this space... 5th time lucky and all that.

Ruby Custom String Delimiters

So, as I'm in the process of transitioning to Bitbucket, which will eventually entail a proper website-ish thing, I'll actually be posting some more developer style stuff here.

First subject: Ruby Custom String Delimiters. This is a feature of Ruby which just shouldn't be allowed. Basically, Ruby claims it can substitute an arbitrary symbol for a string delimiter, by prefixing the delimiter with %q or %Q (depending on if you want a double or single quoted string, which are different in Ruby). Also using a symbol which has a matching closing symbol will cause the closing symbol to be the end of the string (e.g. using [ causes ] to end the string).

Got that? Well, here's some counter-intuitive stuff you have to deal with when trying to parse the damn thing, when playing around with IRB and Ruby 1.8:

•  %q"Hello\n" == "Hello\n" => False (so it's insufficient to look at quotation mark to determine type of string)
• %q][] does not end the string
• %q][]] is a syntax error
• %q£Hello£ quits IRB  

I just can't conceive of a circumstance where this feature is actually useful, outside of obfuscated code competitions. I've programmed in a fair few languages, and I've never once thought this would be useful enough that it outweighs making the code harder to read. Even if it's inherited  from another language, this is a good example of why I don't like Ruby: too much flexibility in the language can be counterproductive to understanding the language.

RPGMaker Trans v2.06 released, now open source

Hey, I actually managed to keep one of my self set arbitrary deadlines. RPGMaker Trans is now open source, and hosted on Bitbucket. So, noticeable changes:

1. Bug reports should now go on the Bitbucket issue tracker - much more useful, and it has the same sign in requirements as the blog, so not any harder to use (well, probably).
2. Downloads are now served by Bitbucket instead of Dropbox.
3. RPGMaker Trans now has an icon.
4. RPGMaker Trans license changed to GPLv3.
5. Technically, RPGMaker Trans is now fully cross platform. To run on another OS, you need Python 3, PySide and BeautifulSoup4, and then run rpgmakertransmain.py in the source. The value of being cross platform is questionable, given that RPGMaker is only on Windows.
6. If running RPGMaker Trans from the source distribution (emphasis: not the binary download), you can now try the undocumented and probably untested CLI (command line interface; useful for people who want to script the translation process). Start RPGMaker Trans with the --cli option and you'll get help on what the parameters should be. This will be added to the readme soonish, but I only just remembered about this, and I'm not going to put off launch to document it.
7. And finally, the blog gets a new theme. It's a suitably momentous occasion.

 Enjoy. I've still got a few more things to push into the repository (e.g. my initial attempts at VX support), but this should be enough for a good enough release.

EDIT: I initially put the wrong binary on the website. If you're one of the 10 who downloaded it and get the error "TypeError: Type str doesn't support the buffer API", please redownload.

Starting to move to open source

Well, it's been a long time coming, but soon (hopefully within the next month or so), RPGMaker Trans will be open source. Good new for those who care about this sort of thing, which probably isn't many of you.

There's been a lot of issues moving to open source; the biggest practical one was that early on I had some help from a real-life friend of mine who did not (and still does not) want to be identified. Practically, that meant that I wasn't able to release his code as open source, and without that, RPGMaker Trans would mostly not work. However, I've recently got in touch with him and he is now satisfied that his code is no longer present in the current RPGMaker Trans codebase, and hence does not object to it being open sourced.

So the current stuff that needs doing before open sourcing:

1. Documentation: Full documentation won't be available until some way down the line, but at least partial documentation needs to be there if open sourcing is to have any value. Module level documentation is now done, and for the first open source release this will do.
2. Deobfuscation: Some code is... slightly obfuscated. No real reason for it, but it needs to be fixed.
3. General tidy-up: I don't think I'll be following pep8 too strictly (for those who know it), but the code is really ugly in places. A quick spring clean would be nice. I ran autopep8 over the code and I think I'll call it a day. Some pylint based stuff will happen later, but not now.
4. License: I've not yet decided on the license. I'm leaning towards GPLv3, but I've not yet decided. Decided that the license will be GPLv3.
5. Project Logo: I need one of these. I used to have an icon for RPGMaker Trans, but it was rubbish... I've got a logo now. Still pretty bad, but honestly, there's only so many ideas for a 'logo for translation software'

 And for posterity, a list of things that have already been done:

1. Removing code that cannot be distributed openly
2. Migrate from Bazaar to Mercurial for version control
3. Setting up hosting at Bitbucket
4. Saner source organisation