Transformer blog

Converted over the dpr and the main application form. Also brought in a stripped-down version of the old SystemFunctions.pas, and did a huge amount of rewriting of the file i/o functions in the application. I'm also using the SelectFileEncoding functions and dialog box I wrote for the Apparatus application when loading files. Transformer 3 now builds and runs, but of course there's a lot of testing to do, and many issues where character encoding will need to be carefully checked. I also have a problem with the main menu having disappeared; it's there, but it seems to be covered up by other components for some reason. Still working on that.

Next link in the chain...

Began focusing on the more Transformer-specific libraries for porting to Delphi 2009, specifically the ReplacePair form/unit, and then the TransformItems library, which defines the core classes of transformation items (currently only TReplaceItem and TScriptItem, but soon to include TXSLTItem).

The ReplacePair form was pretty straightforward, but there was one problem with TPerlRegEx, which I'm using to replace TURESearch. Compilation would fail with a fatal error, for no discernible reason. I eventually tracked down this bug, and was able to use the workaround (adding a pointless call to the Match method).

Moving into TransformItems.pas I started to hit the inevitable problems that come with using the open-source JavaScript Bridge library, which has lain untended since early 2004. I had to make one tweak to js15decl.pas, to overcome an ambiguity with StrDispose, which now has two variants, one for PAnsiChar and one for PWideChar. I chose the latter version, so it now compiles, but I have no idea whether it will work or not. However, I did also find this project: (http://code.google.com/p/rawfpcjs/) (blog is blocking Google URLs in links at the moment), which is very recent, and looks promising; if there are problems with the use of JSBridge in D2009, I can probably move to this alternative, and it might even make life simpler. There's still a lot of work to do on TransformItems, and then on the individual ScriptItem units, but I'm getting close to working on the actual application, finally.

Added the code to map the old output filename variables onto the new template system. The Batch window functionality is now complete.

Of the three items listed in the previous posting, I've fixed the menu positioning issue, and I've cleared out the old unused message TStaticText controls. I've also made a start on a version conversion system for batch files, by adding a version element to the output file, and checking its presence/absence/value when loading a file. Now I just need to add the code for converting the complicated old settings (5 variables!) to the new single OutputFilenameTemplate string value.

Finished the output filename mask code, with the live demo of the mask in action and the file i/o fully tested. A couple of things remain:

• That popup menu is still in the wrong place.
• I haven't decided how to handle old Transformer files; if it's even possible to map the old setup onto the new, I should try it, but I should also warn people when they open an old file that they should review the filename settings.
• I probably haven't cleaned out all the obsolete messages stored in labels in the Batch form.

This stuff shouldn't take long, and then I'll be in a position to take on Transformer itself, starting with the data structure for replace items.

As part of rewriting the Batch file processing screen, I've been looking closely at the clunky old system by which the results of files were saved to a designated location and name. I'm going for a placeholder-based system similar to that used in oXygen's transformation scenarios, but a bit more flexible (it has more human-readable placeholders, and has one for the original file extension). This also integrates with controls for choosing an actual folder as part of the filename. I have most of the work done, but there are some GUI issues to fix -- I need to add a plain folder image to the Nuvola dll, add images to the popup menu, and figure out a problem with the position of the display of the popup menu, which is currently in the wrong place. I've also refactored a lot of the original code, renaming components to remove the leading "u" which was used to designate a TTntUnicodeControls component.

A great addition to the application, so I've rebuilt the installer (same version, of course -- there's no change to the executable), and updated the documentation and the Web site. I've also updated the roadmap and future features information to show my current plans for the application.

I found this excellent implementation of the PCRE library wrapped for Delphi by Jan Goyvaerts. The Delphi code is MPL 1.1, and the PCRE engine is BSD, so it's all usable in any of our projects, and it's perfect for Transformer because it handles UTF8. I initially compiled and installed the component, which is designed to work with the dll that's shipped with it, but every time I destroyed a created instance of the component, I was getting access violations, so I edited the source to link to the C object files instead of linking to the dll. This uses a slightly newer version of PCRE, and more important, it doesn't generate the AVs. Built a test app, wrapping the component in my own class to suit what Transformer will want to do. Everything works fine!

So yet another mighty step forward: now I can provide good Unicode regular expression support, without having to map it to the JavaScript library.

Made fantastic progress today. This is basically what I've implemented:

1. Several LoadFileToString functions with a range of different input parameters.
2. Functions for detecting the character set on load, including peeking into XML and HTML headers, and detecting UTF-8 byte-sequences.
3. An inventory (TDictionary) of code pages known to Windows, which make it possible to look up any code page id found in a header.
4. A dialog box which will let you test any of the different code pages against your text until you find the right one, with live conversion and font control.
5. Lots of testing with a variety of languages, encodings and BOMs.

With the exception of UTF32, I now have all of this stuff working. I'll have to add the UTF32 handling, and then work on finding a decent open-source implementation of regular expressions for Delphi. At some stage, it might be worth trying to take the broken port of Mozilla code, which has functions for recognizing likely ANSI encodings by their byte sequences, but that might be overkill.

This really has been hard, but quite rewarding, and infinitely valuable. I can add to Transformer the ability to specify an input code page as well as an output encoding.