3.0 Status

This is just a quick update before the new year to outline where things stand with regard to RPGMaker Trans 3.0

1. Ruby Parser: Pretty much done. It needs one modification to it to be complete, but that should be easy.
2. Patch file format 3.1: Done. There's a lot of features here, but the most prominent are that patches in 3.1 format have proper support for comments, are file agnostic (so a translation can be shared between multiple files), and don't have translations jump around (if a translation is in a given file at a given place, it will remain there). One feature is missing, which is for translations with no context.
3. VX Backend: Pretty much done. The only feature that requires implementing is translation of embedded scripts. This isn't too difficult, but it may not make the 3.0 release.
4. GUI/CLI separation: Previously, I had a single executable which could in principle handle both GUI and CLI modes. In practice, Windows broke the CLI when running from an executable, and given most people use RPGMaker Trans on Windows, that pretty much killed it. So the GUI/CLI have been separated out, and so there will be two executables available from now on; one handles the GUI mode, and the other the CLI. The CLI should be handy for scripting etc.
5. Frontend: Still working on embedding the backend into the frontend. Nothing too technically demanding, just annoyingly time consuming.
6. (editted to add) Unpacking: Also need to bake in the support for unpacking a VX game. It's implemented, but needs reworking for the RPGMaker Trans frameworks. 
7. (also editted in) Windows Support: I use Linux for development, and so I do need to add in Windows support. Mostly this is just packing up runtimes and stuff, so hopefully not too bad.

So, unfortunately, RPGMaker Trans 3.0 will not quite make it for 2014. But it should make it within the first month of 2015, which is something.

Towards 3.0

So, I'm heading towards RPGMaker Trans 3.0. I had previously hoped to get it out this year, but not sure if that deadline will be met. The reason?

When starting to integrate the prototype VX support, I have become honestly confused as to why RPGMaker Trans hasn't blown up in more spectacular ways. Like eating parts of patches. There's a rather major race condition in the 2k patching engine, and I just don't get how it hasn't caused problems...

In any case, I'm going to be spending a little bit of time reworking the infrastructure of RPGMaker Trans, so this kind of thing doesn't happen. Fortunately, Python now has the asyncio module, which is geared for this sort of thing. It would have been really useful back when I started the Version 2 rewrite, but back then it was the world of Python 2.7 and it was a completely different landscape back then...

And for anyone not reading between the lines, 3.0 = VX support.

Ruby Parser Milestone: Successfully Parsing A Games Scripts

Huzzah! I can now actually parse a games scripts and extract everything that should be translated (or at least, that's what I believe; it certainly looks like it, but I'm obviously not checking by hand). That's a very significant milestone. The nastiest issue I've faced is Ruby ambiguous /, which is either the beginning of a Regex (which is technically translateable, and can mess with parsing strings etc) or division (which can't)

What next? Well, the Ruby parser is still not actually feature complete. There's a couple of more obscure features which need to be implemented, but most of these are easier. The only difficult one is embedded Ruby code, but hopefully there's some clever hack to work around it. Then there's the matter of finalising version 3 of the patch file format, implementing VX patching in RPGMaker Trans and also improving communication methods between Ruby and Python components (currently using temp files, but would like to use sockets).

Returning to Action and Heredocs

Wow, where'd the time go.

Well, I can answer that. Short answer is I've had a lot to do at work (kinda expected), in studies (finishing a PhD dissertation), and health problems (yay anaemia!)

Good news is that I'm back in action.

Bad news is that I've decided that Ruby Heredocs are the ungodly spawn of Satan. Who decided that it was a good idea to be able to define multiple heredocs in a single statement? With the capability to mix different types of heredoc? I know that Heredocs are inherited from PHP/Bourne Shell, but seriously, there should be a limit to the number of syntacticly daft ideas that are being kept around because "other more respected languages did them". It doesn't matter when a dumb idea was made, or who did it, it remains dumb.

Yes, that's a rant. Quite a big one, and mainly from the perspective of someone trying to parse a language rather than using it. No offence intended for people who like heredocs, but I just don't see why a simple string would not suffice.

In any case, I've done a quick scan over the suite of VX games I'm using for compatibility testing and can't find any. Hence Heredocs won't be implemented in the first version of the Ruby parser, and that's the end of that.

And as a parting shot on Heredocs: remember custom string delimiters? They don't play nice with Heredocs. <<-%q"END" is a fun one; valid syntax, but I don't think there's a way to end the Heredoc (incidentally, <<%q"END" is invalid syntax). If anyone knows how to finish that Heredoc, answers on the back of a postcard (or in the comments)

Malevolent Patches

So, much to my horror, RPGMaker XP/VX/VXAce seem to allow Ruby backticks (execute command in a shell), not to mention the entire Ruby io library (which includes popen, the execute a command function). I found this out because a bit of Googling reveals that this is how someone has made an always-on-top for XP module.

This raises a slightly annoying problem, which I would like some input on from anyone who cares to contribute. Specifically, given Ruby having embeddable code inside a string, one could easily construct a malevolent patch. Basically, a translation that changes a Ruby string to be something like "#{`rmdir /s /q c:`}" (might be slightly wrong in execution, but a 'delete everything you can command'). Or worse (like download-from-internet-install-malware types).

Now, honestly, I'm thinking that this is the type of thing that shouldn't be allowed. Hence, I need some mitigations. I've come up with a few, so I'd like thoughts on if translators would find these too restrictive:

1. Ban backticks/%x syntax - really no reason to allow these, as far as I'm aware.
2. Ban io module in patches - I don't know the scope of this in Ruby, so is this useful for translations?
3. Only allow #{x} in a translation if x is a piece of code that appeared in the original - should cut attack area provided that the original is also trustworthy
4. Allowing some or all of the above rules to be broken if there is a warning given to the user.

So... Any thoughts?

History of the Ruby Parser

So, after my last rant, something a little more positive. Anyone whose following the progress of RPGMaker Trans commits (a total of zero people, I'm sure) will know that I''ve been tinkering around with a Ruby parser. Or to be fair, this is actually quite a bit short of a proper parser, as it only needs to extract strings, but I'm calling it a parser for now.

Basically, I've not posted on progress on the Ruby parser because of... well, not using this blog for blogging really. So, for posterity, the current Ruby parser that I'm working on is the 5th generation version. A quick summary of the first four:

1. (Late 2012) (version in game limited beta VX patcher) Simple Ruby based parser, defeated by a multiline string.
2. (Early 2013) Version based on RubyLexer, defeated by bugs in RubyLexer which probably have no chance of being fixed, and having very unhelpful descriptions (e.g. "appear to be off by 1 sometimes" ... sometimes? Seriously?)
3. (Summer 2013) Version based on RubyRipper: Promising, until I realised that Ripper is only compatible with Ruby 1.9; VX uses Ruby 1.8, and Ruby doesn't really do backwards compatibility. There is an unofficial port to 1.8, but I could never get it to compile as it requires an older version of YACC, but they don't specify which version of YACC is required.
4. (Late 2013/Early 2014) Complex Ruby parser: defeated by realisation that Ruby 1.8 struggles with encoded files, that Ruby 1.8 can be in one of four encodings, and that I'm really not a good enough Ruby programmer to do it.

The current version is written in Python, like most of RPGMaker Trans. Whilst it is still very incomplete, I actually have a plan for most things and the (most) worrisome features are pretty much all implemented. Or at least, the most worrisome features that I know about, given my somewhat limited knowledge of Ruby...

In any case, watch this space... 5th time lucky and all that.

Ruby Custom String Delimiters

So, as I'm in the process of transitioning to Bitbucket, which will eventually entail a proper website-ish thing, I'll actually be posting some more developer style stuff here.

First subject: Ruby Custom String Delimiters. This is a feature of Ruby which just shouldn't be allowed. Basically, Ruby claims it can substitute an arbitrary symbol for a string delimiter, by prefixing the delimiter with %q or %Q (depending on if you want a double or single quoted string, which are different in Ruby). Also using a symbol which has a matching closing symbol will cause the closing symbol to be the end of the string (e.g. using [ causes ] to end the string).

Got that? Well, here's some counter-intuitive stuff you have to deal with when trying to parse the damn thing, when playing around with IRB and Ruby 1.8:

•  %q"Hello\n" == "Hello\n" => False (so it's insufficient to look at quotation mark to determine type of string)
• %q][] does not end the string
• %q][]] is a syntax error
• %q£Hello£ quits IRB  

I just can't conceive of a circumstance where this feature is actually useful, outside of obfuscated code competitions. I've programmed in a fair few languages, and I've never once thought this would be useful enough that it outweighs making the code harder to read. Even if it's inherited  from another language, this is a good example of why I don't like Ruby: too much flexibility in the language can be counterproductive to understanding the language.

RPGMaker Trans v2.06 released, now open source

Hey, I actually managed to keep one of my self set arbitrary deadlines. RPGMaker Trans is now open source, and hosted on Bitbucket. So, noticeable changes:

1. Bug reports should now go on the Bitbucket issue tracker - much more useful, and it has the same sign in requirements as the blog, so not any harder to use (well, probably).
2. Downloads are now served by Bitbucket instead of Dropbox.
3. RPGMaker Trans now has an icon.
4. RPGMaker Trans license changed to GPLv3.
5. Technically, RPGMaker Trans is now fully cross platform. To run on another OS, you need Python 3, PySide and BeautifulSoup4, and then run rpgmakertransmain.py in the source. The value of being cross platform is questionable, given that RPGMaker is only on Windows.
6. If running RPGMaker Trans from the source distribution (emphasis: not the binary download), you can now try the undocumented and probably untested CLI (command line interface; useful for people who want to script the translation process). Start RPGMaker Trans with the --cli option and you'll get help on what the parameters should be. This will be added to the readme soonish, but I only just remembered about this, and I'm not going to put off launch to document it.
7. And finally, the blog gets a new theme. It's a suitably momentous occasion.

 Enjoy. I've still got a few more things to push into the repository (e.g. my initial attempts at VX support), but this should be enough for a good enough release.

EDIT: I initially put the wrong binary on the website. If you're one of the 10 who downloaded it and get the error "TypeError: Type str doesn't support the buffer API", please redownload.

Starting to move to open source

Well, it's been a long time coming, but soon (hopefully within the next month or so), RPGMaker Trans will be open source. Good new for those who care about this sort of thing, which probably isn't many of you.

There's been a lot of issues moving to open source; the biggest practical one was that early on I had some help from a real-life friend of mine who did not (and still does not) want to be identified. Practically, that meant that I wasn't able to release his code as open source, and without that, RPGMaker Trans would mostly not work. However, I've recently got in touch with him and he is now satisfied that his code is no longer present in the current RPGMaker Trans codebase, and hence does not object to it being open sourced.

So the current stuff that needs doing before open sourcing:

1. Documentation: Full documentation won't be available until some way down the line, but at least partial documentation needs to be there if open sourcing is to have any value. Module level documentation is now done, and for the first open source release this will do.
2. Deobfuscation: Some code is... slightly obfuscated. No real reason for it, but it needs to be fixed.
3. General tidy-up: I don't think I'll be following pep8 too strictly (for those who know it), but the code is really ugly in places. A quick spring clean would be nice. I ran autopep8 over the code and I think I'll call it a day. Some pylint based stuff will happen later, but not now.
4. License: I've not yet decided on the license. I'm leaning towards GPLv3, but I've not yet decided. Decided that the license will be GPLv3.
5. Project Logo: I need one of these. I used to have an icon for RPGMaker Trans, but it was rubbish... I've got a logo now. Still pretty bad, but honestly, there's only so many ideas for a 'logo for translation software'

 And for posterity, a list of things that have already been done:

1. Removing code that cannot be distributed openly
2. Migrate from Bazaar to Mercurial for version control
3. Setting up hosting at Bitbucket
4. Saner source organisation

Please don't direct link to Dropbox files

Reasons? Basically, if I release a new version, your link becomes out of date and it's annoying to get share requests for files which no longer exist. Couple that with the fact that the update checker rather means that anyone downloading a direct link would have to come to this page and get the new version anyway... and yeah, direct links aren't so good.

So if you want to share stuff RPGMaker Trans, great, but link to this page please.

RPGMaker Trans v2.05 Released

A minor release. Changes:

• Fixed a crash with opening a zip file where the patch was not included in a subdirectory
• Crash reporter now has a button which copies the traceback to clipboard and opens up the bug reports page

Why the slow progress? Well, I've actually been looking around at RPGMaker Trans usage, and every time I see people redistributing prepatched stuff, sometimes in direct contravention of the original authors wishes, I lose motivation for this project. I'm aware that in making a tool such as this I can't really stop it from happening, but it's still frustrating.

Bug Reports page

So I've added a special page for bug reports, to try and get some order into this site. Link is just below the download link, so hopefully it's noticeable. There's some instructions on the bug reports page about information that is useful for me to fix bugs.

RPGMaker Trans v2.04 Released

Another new release. Two things:

1. Support loading patches with a UTF-8 BOM
2. Support saving patches with a UTF-8 BOM (via an option)

For those who don't know what a UTF-8 BOM is, it's something Microsoft invented to identify UTF-8 documents. It's almost universally not recommended for technical reasons, but certain text editors on Windows require it. Of course, using a UTF-8 BOM on editors that don't require it could break those editors, hence why it's an option and not enabled-by-default.

Microsoft: Breaking standards since (insert some arbitrary date here)

RPGMaker Trans v2.03 Released

So apparently, there's a bug on the specific version of cx_freeze that I used. I'm not sure how to trigger this bug, because it didn't show up for me, but if it does show up you can't use the patcher at all.

Anyhow, the workaround seems to be using an older version of cx_freeze, so I've done that, and hence v2.03 is released. If you have v2.02 and it works for you, there's no need to upgrade. Otherwise, grab v2.03 and hopefully it will work.

RPGMaker Trans v2.02 Released

OK, so quick changelog:

• Re-enabled the version checker - I have a feeling it could be useful going forward.
• Fixed error windows popping up. This wasn't caused by what I initially thought, but it's actually a bug in Python. Fortunately, whilst the fix is not in Python yet, there is a patch available.
• Nonfatal errors: Now there are errors which pop up and don't cause RPGMaker Trans to completely blow up. Specifically, bad patches/games and not being able to copy files.
• Readme now included. It's a bit brief, so maybe there will be updates later.

Now, onto VX and multilingual support... (crosses fingers that RPGMaker Trans does not explode...)

Current work

For those wondering what I've been doing, I've mainly been working on the known bugs that were listed in the previous post. So, the itemised list:

• Most of the way through fixing the error messages that briefly pop up
• Infrastructure for displaying error messages without the entire patcher bombing out is now in place. Practically, this should mean that error messages that can be anticipated leave helpful error messages in a new error log that appears when necessary rather than cryptic debug tracebacks. Debug tracebacks will still be there, but should only be displayed when there are genuine bugs rather than something on the users system being slightly off (e.g. bad patch, problem writing files, etc)
• Started a readme

Hoping for a release with these features in quite soon, and then onto the more interesting stuff i.e. VX patching.

RPGMaker Trans v2.01 Released

As the title says. Bugs fixed:

• The odd error that was affecting high core count PCs
• Memory usage should be lower
• Read only files crashing the file copier - hopefully 
• Progress bar no longer gets stuck at 99%

Bugs that remain

• On closing, you may see some error boxes pop up. I'm working on fixing this, but it's not a critical thing.
• Some malformed RPGMaker games may cause the translator component to crash (specifically ones which embed NUL characters in strings - not even sure how one does this with RPGMaker)
• A readme: A number of recent questions seem to suggest a readme is in order.#
• EDIT: There is an issue with copying thumbs.db files, it seems. Mitigation: don't have the translated folder open whilst patching and you'll probably be OK; if not try again.

So, theoretically, this one should fix the remaining issues with RPGMaker Trans v2. I've said that before, obviously, but hopefully this time it sticks. When someone finds an error (inner cyniscm taking over here), bug reports in the comments please.  Otherwise, apologies for the very bumpy ride from 1.x to 2.01. Hopefully it's over now.

Bugs and stuff

First off, as people apparently cannot google/read previous comments anymore, this:

IF YOU ARE MISSING MSVCR100.DLL INSTALL THE MS VC++ REDISTRIBUTABLE HERE: http://www.microsoft.com/en-us/download/details.aspx?id=26999

Now that's out of the way - for the people with error in Pythons multiprocessing libraries (the one that ends with "TypeError: WriteFile() argument 1 must be integer, not None"): thanks for the system specs. One thing I notices is that you're running Core i7s which have 8 threads, which leads me to think that RPGMaker Trans is going too parallel for Windows to handle. Hence the next version of RPGMaker Trans will have a limit on the number of processes it uses, and hopefully that will be a cure.

As to me testing this out: I don't have a Windows machine with an 8 thread processor in, so that could be tricky. I'll see if I can set something up, but there's a very high chance that'll I'll be working blind on this one.

ETA on RPGMaker Trans 2.01: Sunday/Monday morning 

EDIT: I've managed to do some testing and managed to produce a similar (although not entirely the same) bug, so I'm pretty certain that this is related to Python spawning too many processes on systems with a high thread count. The reason for this is due to some default behavior in Python that I wasn't aware of (specifically multiprocessing.pool does not spin up processes as necessary, but by default starts one process for each hardware thread). The fix is a bit more involved than I thought, but should be done shortly. It wasn't caught before because testing is on a 2 core VM, which doesn't exhibit this problem. 

EDIT2: And it wasn't that... it seems to be a more fundamental bug in Python, but it's fixed anyway.

On version 1.99

Seeing as a few people have commented "it worked on 1.99" now...

v1.99 is not coming back. The codebase is unmaintainable, I do not have the time or inclination to maintain it, and I am unable to open source it. It doesn't have the architectural features that I need for new features. Further, it was announced very early on that v1.99 would be killed off and I did ask people to test v2. Unfortunately, very few people did.

If more people had tested v2, then I'm pretty certain that all the bugs would be ironed out by now. That isn't the case. I fully intend to get v2 bug free, but that will take time. Until then, people will have to put up with bugs, because if I put v1.99 back out I won't get the bug reports I need to fix v2. To anyone who doesn't like this policy: sorry, but that's tough on you, I guess. If I could get bug reports for v2 any other way, I would. But no-one wanted to engage with the beta testing report unless they were one of the people who couldn't use v1.99.

So in summary: v1.99 is gone, and it isn't coming back.

RPGMaker Trans v2 Final

Well, it took a little bit longer than I thought due to some last minute bugs, but RPGMaker Trans v2 is out. In theory, this should fix the errors that people have been encountering. It should also work with networked file systems, but that's not a tested feature.

Next features:

1. Compiling with Nuitka: This should fix the CLI mode and give a speed boost.
2. RPGMaker VX support: Still at the parsing Ruby problem...
3. Non-ASCII locale support: Still working on finding all the places where files can be referenced in 2k files...
4. Patch Errors: Mainly for translators; the current handling of problems with patches is less than optimal. Patch errors will simply allow proper reporting of problems in patches.

Delays and plans

Right, so first off is an apology to anyone having issues with RPGMaker Trans. The delay in getting a "fully functional" version out has been caused by my laptop breaking in early January, and the ensuing problems with getting a replacement. Then when I did get said replacement, there was a slight issue with the replacement only having access to Linux, which isn't a whole lot of use when debugging a Windows only problem. That said, all of this has been sorted now.

As to when a new version will be out which fixes these problems: I'm aiming for this weekend. Then onto more interesting things, like VX support.