Transformer is a great tool for working on Unicode texts, but today I hit a problem, in that I needed to work on Hot Potatoes data files, which are not actually Unicode; they're 8859-1 files with all upper-ascii characters escaped to numeric escapes.

It seemed a shame not to be able to work on those files, since the actual underlying format is Unicode (they use Unicode characters, but encode them as numeric entities). So I made a couple of changes to Transformer to make that possible. First, I analysed the file load routines: there are two, one for the source text in the main screen, and one used to load files during batch operations. These were both failing to load plain ascii files, because they appear to be UTF-8 with no BOM, but they're not. So I added a couple of lines so that, in the event of a failure to load a file as UTF-8, it will be loaded as plain ASCII and then turned into a WideString.

Secondly, I needed a way to save a file as ASCII or ANSI, but deal with any characters over 127. I added an option to the batch window to save as "ASCII with numeric entities", which escapes all characters above 127 to HTML-style numeric entity references, and then saves as ASCII.

This all seems to be working well, but it needs to be documented, and the same save function should also be added to the output text save dialog and routine, for symmetry. Adding this as a task so that I get around to doing that.