Academic

I've added case normalization and punctuation stripping to my USM library, on the basis that ShingleCloud basically does both of these things, so the comparison between them is not really level unless this is done. The results will most likely be to even up the runtimes, because USM will run more slowly, but presumably it will improve the "success" of USM when compared to SC, which is currently doing better (based on my own human judgement of successful matches). I'll now have to re-run all the tests again with this new measure, but I think what I'll do is to merge all of the different variants for the Lear texts into a single operation, which will take a long time to run and process, but will perhaps give more useful output when it's done.

I've also done a fair amount of examination of the existing Lear results, and it does seem that without normalization, even SC needle/haystack calculations result in a longer run of intuitively "good" matches before spurious ones begin to appear. USM/NCD manage about 60 matches before we start to see oddities, but SC gets into the 70s.

Found a good article on the limitations of NCD which result from the block window size of the various compressors one might use to do the NCD calculation; this points out that if the length of the concatenation of the input strings exceeds the block window size, then NCD is basically useless. The limit for bzip2 (in default configuration) is 900KB, and for GZip, which I'm using, it's 32KB. This underlines my conclusions that USM deployed on text is best suited for short strings, which is the precise context in which other metrics become less powerful because of their tokenization strategies.

Figured out the problem with the large-scale Lear test script using Jaccard, and I'm now running that test (it takes a while). Following this, I'll create new versions of all of the scripts using character tokenization and Jaccard, and see how those go. That should give me a good all-round view of NCD vs SC. NCD (Complearn) uses bzlib by default, and I'm using GZip, so this may account for the differences between them.

Started early, worked late to finish XSLT transformation for tomorrow.

Started at 8.30 this morning.

The oddity I saw yesterday was a bug, and AM kindly fixed it overnight, so I've run most of the tests again using the new version, and got some interesting results. So far, I've run the three shorter tests successfully with word tokenization and ngram=1; SC now seems to be tracking USM more closely, but is still less granular on shorter strings. I need to do this again with character tokenization. The last, longer test with 5200 comparisons didn't work properly -- SC gave me 1 for every result -- so I think the script needs fixing, then running again.

AM created a new version of the SC library which has the capability of doing Jaccard Coefficients, so I've been writing new versions of my scripts to see how those numbers stack up against mine. Right now I'm still not sure how to convert them to a scale comparable with NCD; I should be subtracting them from 1 to get Jaccard Distance, which is analogous to NCD, but some of the results are greater than 1, so result in negative numbers.

There most of the day. Long trek across campus and back, twice, takes time too, but it's probably good for me in some way.

I'm helping with an unusually large DHSI course on text-encoding all this week.

• Tweaked all my scripts and re-ran all of them.
• Added scores from the command-line compiled NCD utility from CompLearn in addition to my USM.
• Wrote XSLT to convert Lear result set into useful tables.
• Rebuilt all the graphs in my ODS files.
• Found what seems like a bug in SC, and wrote to SM with details.
• Wrote the outline to my presentation

There's still lots of work to do to get the presentation itself written, along with my speaking notes, but I think I'm pretty clear on what I want to say now.

MB sent me some candidate pieces of text for comparison, and for my final array of data, I decided to use a piece from the last act of Lear. I've prepared the lines (65 in the quarto, and 80 in the folio), and written what turned out to be quite a complicated bash script to do the following:

• Starts writing the output as an XML file.
• Reads the lines of each source into a pair of arrays.
• Writes out the lines with suitable numbering as XML, so that they're available in the output file (which I intend to process using XSLT).
• Compares each line in each array to every line in the other array (5200 comparisons), using both USM and ShingleCloud.
• Writes out the results to the XML file.

The complexity largely arose from having to learn aspects of bash scripting that I didn't know, including the oddities of using arrays, and variable scoping. My intention is to sort the results according to descending order of USM similarity, and SC similarity, and compare the order, to see if they're both doing the job in a comparable way, and to see where the distinctions lie. I also plan to build timing into my bash script so I can see which is quicker, and by how much.