Transformer blog

Some time ago Stew asked me to do some work on a jwplayer instance for the Thomson mystery. I did the work then, but we recently needed to make some basic changes and hack the surrounding page to make things work in the site.

Using the new v3 for work on the Arabic writing site, I found a bug whereby the trailing backslash was missing from the backup folder, causing files to be backed up one folder up in the tree. Fixed the bug and tested successfully.

JJ reported a problem with the site today. The eXist service wasn't running so I started it, but the experimental map wasn't running. Toolbar and so forth appear but no map.

I've tailed the access and error logs and find no errors.

Using 3.0 to help mark up a ScanCan doc, I noticed that changing the Checked Only checkbox didn't set the Modified flag on the current file; fixed that, and then fixed a bug resulting from that fix. Also added a new "secret" Action invoked by F9 which takes content directly from the clipboard, transforms it, and places it back on the clipboard, also showing it in the GUI (there was a way to do this without the GUI before, but when you don't see anything happen, it's a bit difficult to know if you hit the right key or not).

Trying to use version 3 of Transformer for marking stuff up in ScanCan, I hit a bug whereby the first time the app starts (when the layout XML files are not there, so it's starting from zero), the resize code which ensures that columns in the main TListView are not resized out of existence or out of view does not work properly, because the initial values for column widths are not readable, or not read. Added a get-out clause in that resize routine so it detects that situation and does nothing; that allows the app to start up with columns not too badly set up, and thereafter you can resize normally, and it will save and read back correct column size values. That makes Transformer 3 usable on Wine.

Working with Transformer in my workshop today, I discovered that the JavaScript engine, which does some automatic code-checking and gives you an error message when your JS code is wrong, was giving me an error message in garbled Chinese. This was doubly confusing because I was actually working on Chinese texts at the time. It turns out that the JavaScript Bridge code I'm using, which was written well before Delphi 2009 was on the horizon, has lots of PChars in its code. In D2009, a PChar is a PWideChar, but the text is actually not a WideString, it's just a plain Ansi string. Rather than hack the JS Bridge code, which is a bit fragile and unmaintained, I just cast the "PChar" return from the JS engine to a PChar and then to a string: string(PAnsiChar(message)). This seems to do the job. The error messages will always be in English anyway.

I also discovered today that my "is it UTF-8?" detection code gives a false positive on GB18030 files (PRC Chinese encoding). This is annoying, but there's not much I can do about it. It just means that when opening such files in Transformer, it always suggests UTF-8 instead of GB18030.

Since I now have robust text file encoding conversion available, I think it would make sense to make this a batchable feature. I'll need to think about how best to do that, but it could be a part of the main batch screen, where you could specify the input encoding. There should be a new, initial tab entitled "Loading files", which should have a drop-down where you can specify the input encoding. You could discover this using the normal file-loading capabilities in the Source tab.

Fixed a bug I found in Transformer when I was using it to prep some texts in the workshop yesterday.

Wrote the back-end code to check input, run XSLT transformations and save the results; added appropriate error messages to the error-logging code for when failures occur, and did some basic testing. Everything seems to be working. This basically means the application is code-complete. Next:

• A serious comprehensive test with a big batch of big files, including Unicode. Encoding is predicted to be potentially problematic; we need files in more than one original encoding.
• Updating of the Help system.
• Building a new installer for a beta release.
• Adding the beta to the website.
• Emailing users who might test it.
• Final release.

Completed all the dialog box functionality for the new XSLT transformation item type. Created a couple of new icons and added them to nuvola.dll, one for an XML document and one for adding an XML document; the latter is used for "Add new XSLT Transformation Item" in Transformer 3. Spent a little time making the file paths for external XSLT files robust; I'm storing both a relative and absolute path for referenced external XSLT files, so that if the sequence file is moved, and the XSLT file is moved relative to it, the absolute path can be reconstructed on load. This is working well.

Now the only thing left to do is the integration of the actual XSLT transformation, using the xsltproc dll from libxml2. This code can be adapted from the Image Markup Tool, and the adapted code will then be re-used in the IMT version 2 when I write it.

I've added the new item type to handle XSLT transformations, and written all the i/o code for it. I've started work on the dialog box for entering and editing that data, and it shouldn't take much longer to finish it. I've also done a bit of testing with the existing code, and confirmed that both PERL RegEx replacements and JavaScript transformations work well, with Unicode characters embedded in them (I used Japanese for testing).

The testbed for this app will probably be the Perseus dataset, which is in an old version of TEI (P4 or P3, not sure which). We'll want to massage it into P5, and at the same time add some extra markup and do some fixes. The data will then be used for the GRS myth-mapping project.