More work on similarity
MB sent me some candidate pieces of text for comparison, and for my final array of data, I decided to use a piece from the last act of Lear. I've prepared the lines (65 in the quarto, and 80 in the folio), and written what turned out to be quite a complicated bash script to do the following:
- Starts writing the output as an XML file.
- Reads the lines of each source into a pair of arrays.
- Writes out the lines with suitable numbering as XML, so that they're available in the output file (which I intend to process using XSLT).
- Compares each line in each array to every line in the other array (5200 comparisons), using both USM and ShingleCloud.
- Writes out the results to the XML file.
The complexity largely arose from having to learn aspects of bash scripting that I didn't know, including the oddities of using arrays, and variable scoping. My intention is to sort the results according to descending order of USM similarity, and SC similarity, and compare the order, to see if they're both doing the job in a comparable way, and to see where the distinctions lie. I also plan to build timing into my bash script so I can see which is quicker, and by how much.