HCMC Blogs

JMeter proved to be the most feature-rich test platform for our purposes, so I've created a test scenario that loads the server much more heavily than we expect it to be peak-loaded in the event of a publicity splash. These are the details:

• Simultaneous threads: 20
• Ramp-up period: 20 seconds
• Loops per thread: 20
• Requests per thread: 3 (one home page, one search, and one doc retrieval with search hit highlighting)
• Total page requests: 1200
• Time to complete: 4 mins 25 secs
• Longest wait: 12 seconds
• Average: 4.25 seconds
• Mean: 4.5 seconds
• Throughput: 271 page views per minute

Admittedly this is a very simple scenario, and if Cocoon is doing any serious caching then that would factor into the situation; but in the case of XQuery pipelines (two out of three of the page views), Cocoon is not caching because it can't know that the results will be similar. Give the fact that this tests gives us nearly three times the peak throughput that I believe we need to plan for, it's very promising indeed. I will develop a more robust and varied stress test, and throw that at the system this afternoon; but for the moment, it looks as though the existing system will be more than capable of handling our publicity phase.

Link: http://www.nytimes.com/2008/02/04/technology/04link.html?_r=1&emc=eta1&oref=slogin

Thanks to colleague Kim Blank for forwarding me this article from the New York Times that looks at the respective websites of Obama and Clinton, suggesting the websites are a political replay of those Mac/PC ads.

According to design experts, the candidates have left a clear blueprint of their personal style — perhaps even a window into their souls — through the Web sites they have created to raise money, recruit volunteers and generally meet-and-greet online.

Sysadmin put me on to BadBoy and JMeter, which they use for load testing. Both look excellent, with BadBoy the most user-friendly in that it provides useful reports; JMeter reports are a bit more complicated to configure, but might be worth the trouble. I'm going to get familiar with these over the next few weeks.

Changes to the processing of notes last week broke the bios page (where the bio content is in note form). Added a couple of tweaks to the XSLT to exempt <note> elements inside <person> elements from normal note handling.

In the process, I noticed that SM's addition of reference elements to the bios is a bit scrappy in its use of punctuation and spacing, so I fixed those up.

If, like me, you're one of those people who can recite lengthy poems and prose passages from memory-- thanks to the rigors of an unreconstructed education--rejoice! A new study has found that it may be possible to train people to be more intelligent, increasing the brainpower they had at birth, and the key is carefully trained memory.

Read more »

When we go for a publicized launch in the Fall, we'll need to be ready for a significant rise in hit traffic on the site. This is obviously not really predictable, but we can learn a lot from looking at the statistics for JL's Canadian Mysteries site, over a period when there was a publicity splash. I've retrieved some stats from Urchin for such a period, and these are the results:

So we need to plan for a dramatic increase of between three and four times the usage that we're used to seeing, assuming the site has had some publicity and traffic before. If it hasn't, we can base our predictions on the raw numbers, looking at something like a possible quarter-million pageviews in the week of the splash. Since this is a BC-centred project, these will all be concentrated during the waking hours, Pacific time. If we take a peak day of 60,000 hits, spread over (say) 15 hours (8 am to 11 pm), we're looking at a maximum of 5,000 hits per hour; we'd probably be safe if we plan for 100 hits per minute. This should be fairly easy to simulate. It's not as high as I feared, so it may be that even the development Tomcat setup will handle it.

Got a request today to grab a CBC video for later inclusion in a GCM DVD - similar to the last one. It was not straightforward, though. The original is a wmv file, which is actually a stream. Opening the wmv file one sees this:
<ASX version="3.0"><ENTRY><REF HREF="mms://a924.v23745c.c23745.g.vm.akamaistream.net/7/924/23745/1207237657000/origin.media.cbc.ca/windows/thenational/archive/tomthomson-030208.wmv"/></ENTRY></ASX>

The useful part of this, for us, is:
mms://a924.v23745c.c23745.g.vm.akamaistream.net/7/924/23745/1207237657000/origin.media.cbc.ca/windows/thenational/archive/tomthomson-030208.wmv

I was unable to find anything that just worked. I tried FFMPEGX, mencoder, vlc etc. They all work to one extent or another, but with generally bad results. So I went to my Ubuntu VM, looked through Synaptic and found mimms (MiMMS Isn't an MMS Messenger Sender, it's an MMS stream downloader).

I installed mimms and ran:
mimms mms://a924.v23745c.c23745.g.vm.akamaistream.net/7/924/23745/1207237657000/origin.media.cbc.ca/windows/thenational/archive/tomthomson-030208.wmv
The app created a file on my Desktop called "tomthomson-030208.wmv" which looks just as good as the streamed source.

Now I can transcode it - I hope.

Taking a day off.

In order to run some of the text-analysis operations we've been planning on the database, we need to be able to grab text-only output from the documents, and be able to specify various types of filter on that output. I've been working on that today. These are the main points so far:

• I've written an XQuery file called text.xq which does the extraction, based on input parameters (see below), and returns plain text. The file has this custom eXist option declared in it:

  declare option exist:serialize "method=text media-type=text/text encoding=utf-8";

  That means I can avoid any transformations at all, which is handy.
• I've set up a page here which presents a simple form from which you can choose the options you want.
• These are the parameters:
  • startYear
  • endYear
  • textProse
  • textVerse
  • imageProse
  • imageVerse
  The first two are date filters, but the other four are booleans which decide whether prose or verse from either of the two types of document is included. Other filters behind the scenes ensure that only transcriptional text is returned; editorial notes, metadata, regularizations, etc. are all excluded.

As usual, XQuery hacking took a long time, but it appears to be working well now, and it's faster than I expected. The user will now be able to get the text in the browser, and feed it as-is into an application, or supply the URL of the filtered text query to an online service such as the TAPoR Tools.

According to a report to be published in tomorrow's New Scientist, blogs, maps, photo sites and instant messaging systems were better at providing warnings, help and lists of how individuals were affected than traditional sources.

Read more »