Found a good article on the limitations of NCD which result from the block window size of the various compressors one might use to do the NCD calculation; this points out that if the length of the concatenation of the input strings exceeds the block window size, then NCD is basically useless. The limit for bzip2 (in default configuration) is 900KB, and for GZip, which I'm using, it's 32KB. This underlines my conclusions that USM deployed on text is best suited for short strings, which is the precise context in which other metrics become less powerful because of their tokenization strategies.

Figured out the problem with the large-scale Lear test script using Jaccard, and I'm now running that test (it takes a while). Following this, I'll create new versions of all of the scripts using character tokenization and Jaccard, and see how those go. That should give me a good all-round view of NCD vs SC. NCD (Complearn) uses bzlib by default, and I'm using GZip, so this may account for the differences between them.