HCMC Blogs

Using new templates from UVic communications, started on website for social justice. No modifications to the base colour and image set needed. As site is small, I removed the tertiary nav elements from the leftcol div and the extra bits elements from the rightcol div, but left the two divs in there. I then used the div id=main class=fullwidth.

Filled the pages with the content provided by BC. Sent URL for review by him. Have a static HTML dev site at http://web.uvic.ca:8080/~socjust

Made a couple of small changes to HM's preface article on her instructions, then started fixing some little niggles that have been annoying me:

• The Search function appears on both the regular TOC page and the proofing TOC. In both cases, it ran its search against the regular documents, bouncing back to the regular TOC page with the results. It now works independently for each page, so if you search on the main TOC, you get results from the main collection, but if you search on the proofing TOC you get results from docs in the proofing collection.
• Clicking on the column header sort links in the proofing TOC page also bounced you back from the proofing TOC to the main TOC. That's now fixed.
• Article XHTML output always included a link at the top to the main TOC, even when the document was in the proofing collection. It now distinguishes between the two locations of the document, and gives you a link back to the proofing TOC if the document is in the proofing collection.

I've now completed all four of the reviews initially assigned to me for SDH-SEMI.

I have four reviews for SDH-SEMI 2009. Did the first one today.

Received HM's preface article for vol 40, and marked it up. In the process, I marked up a series of references to the other articles in the issue, using relative links. This threw up a new situation in the PDF export. Previously, I was handling two distinct cases of links in PDF output:

• Those beginning with #, being internal links; easy to process into internal PDF links.
• Those beginning with http, being external Web links, also easy to process into external PDF links.

Now there was a third case:

• Those not beginning with a hash, and not containing a protocol, meaning that they're relative.

Relative links don't work in PDFs because they're typically downloaded to a temp folder and opened from there. It was necessary to reconstruct the full original URL of the request so that I could build a full link, to create a proper PDF external link. The Cocoon {request:requestURI} variable, available in the sitemap, should, according to the documentation, give you the full URI, but it doesn't; it gives you only the path after the port. After some reading around, I found the solution. In the sitemap, you pass this into the XSLT transformation:

<map:parameter name="browserURI" value="{request:scheme}://{request:serverName}:{request:serverPort}{request:requestURI}" />

giving you a variable with the full request path, and then you can do this, to reduce the path to its directory, in the XSLT:

<xsl:param name="browserURI" />
  <xsl:variable name="uriDirLength" select="mdh:lastIndexOf($browserURI, '/')" />
  <xsl:variable name="docPath" select="substring($browserURI, 1, $uriDirLength+1)" />

I'm blogging this in detail because the Cocoon documentation is ropy in this area, and I've previously tried to figure this out and failed.

I've been looking at options for free visualization tools that we might use with CityStats output data. The best I've seen so far seems to be ManyEyes. We'll need to come up with simple CSV renderings of our wizard output tables (and/or the original data table displays) which people can then upload; and I think we could also provide some good example renderings, along with some instructions, to get people started.

Looking for ways to implement automated topic discovery, I've determined that I'll definitely need to do part-of-speech tagging as part of creating input for any system that purports to do topic discovery, so today I've been downloading and playing with various POS taggers. The best seems to be the Stanford Log-linear POS Tagger. With the help of Matt Jocker's excellent tutorial, I got started on POS tagging some of the ColDesp files.

One note on the tutorial: it's aimed at *nix, and it says that you can process the content of multiple XML elements by separating them with a backslash-escaped space in the command line (p\ ab to do <p> and <ab> tags). On Windows, you actually need to use quotes ("p ab").

I've been working so far on the basis that the meat of the content for any of my files is going to be contained in <p> and <ab> tags, so that's what I'm working with. The POS tagger simply outputs the rest of the file unchanged, and tags up the text content of the tags you specify. At the command line, you specify one input and one output file, but we have thousands to do, so I needed to write a batch file. I have one working now; it's taking between one and ten minutes to process a file, depending on the size and complexity of the file. The approach I've taken is to put the source files inside the tagger's directory structure, and put a batch file inside the folder containing the source files; the output folder is parallel with the source folder, and the jar file is a couple of directories up the tree. This is what the batch file looks like:

for %%i in (*.xml) do java -mx300m -classpath ..\..\stanford-postagger.jar  edu.stanford.nlp.tagger.maxent.MaxentTagger -model ../../models/bidirectional-wsj-0-18.tagger -xmlInput "p ab" -textFile %%i > ../postagged/%%i 

meaning "for each XML file in this directory, run the jar file operation with the specified memory and model settings, and output the result using the same filename into the postagged folder".

At an average of 5 minutes per file, it would take us nearly 600 hours to get the whole job done; that's 25 days straight. Matt's tutorial explains how to use XGrid to farm out this kind of job to multiple Macs, but it might be simplest just to set up a single dedicated Linux box and have it churn away for the duration; it would probably go faster if it wasn't doing anything else (as opposed to running lots of foreground apps like my machine is).

My next problem is how to do noun-phrase discovery and analysis based on the POS-tagged text.

We're now keeping track of what queries people generate through the wizard interface, so I've created a little admin page which shows a readout of them, with links so you can look at them, the ability to filter by ip and date, and a button for deleting each one in the list. This should be all we need for managing these records for the moment. It's netlink-protected for privacy reasons.

I've been looking again at the prospects of providing some alternative paths into the collection, or ways of browsing it, other than the search or date-order options. Today I did some more background reading, and came up with a couple of ideas that are worth investigating.

It looks like an approach called scatter-gather would be appropriate. This was new to me. It involves a sequence of:

1. Scatter: present the collection as a series of themes or topic areas, from which the reader chooses one or more;
2. Gather: collect all the documents from the chosen areas into an ad-hoc subcollection;
3. Scatter again: divide the subcollection into another set of themes or topics, from which the reader again selects;

and so on, until you get down to a manageable subset of documents. Typically, the initial scatter on the whole collection is done offline, but subsequent scatters are done on-the-fly, as the reader narrows down their topics.

It seems to me that for our particular collection -- huge, baggy, and with no fundamental organizing principle, where even date order is problematic -- this might be a very effective research or reading strategy. The best description of this I've come across so far is in a 1992 paper, "Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections". A good approach to the Scatter phase is found in "Fuzzy Clustering for Topic Analysis and Summarization of Document Collections".

So now I'm looking for tools to help me do this. I've found lots of papers detailing algorithms for fuzzy topic extraction and grouping, but I can't find any actual working implementation that I can use. I'm not anxious to start implementing anything like this from scratch on my own.

I've written to the TEI list to see if anyone can recommend any implementations or tools. Meanwhile, I'm still reading on the general topic of what seems to be called "Information Foraging Theory". I think one of the principal values in the ColDesp collection, independent of its historical significance, lies in its suitability for just this kind of research into searching, browsing and reading strategies.