In preparation for an elegant search tool for the site, I've begun trying to find simple methods of including scrapers for the Library and Archives Canada site and the Commonwealth War Graves Commission site. This post is just some documentation for how to scrape these sites. The scraping was done in Linux.

Straightforward retrieval of CWGC info: if you know the CWGC id (in this case 922332 - Eric Bowen-Smith) of the soldier use
http://www.cwgc.org/find-war-dead/casualty-details.aspx?cid=922332

Not-so-straightforward retrieval of CWGC info, scraping soldier info from page:

curl --silent "http://www.cwgc.org/find-war-dead/casualty-details.aspx?cid=922332" | tr '\n\r' ' '|sed 's/  //g'| grep -o 'dottedBoxContent"><h2 class="redText">.*<div class="casualty_buttons">' | sed 's/dottedBoxContent">\(.*\)<div class="casualty_buttons">.*/\1/'

Which returns some HTML like:
<h2 class="redText">BOWEN-SMITH, ERIC</h2>
<dl>
<dt>Rank:</dt><dd>Private</dd>
<dt>Service No:</dt><dd>13152</dd>
<dt>Date of Death:</dt><dd>02/06/1916</dd>
<dt>Age:</dt><dd>23</dd>
<dt>Regiment/Service:</dt><dd>Canadian Infantry</dd>
<dt>&nbsp;</dt><dd>"D" Coy. 5th Bn.</dd>
<dt>Panel Reference</dt><dd>Panel 18 - 26 - 28.</dd>
<dt>Memorial</dt><dd><a href="/find-a-cemetery/cemetery/91800/YPRES (MENIN GATE) MEMORIAL">YPRES (MENIN GATE) MEMORIAL</a></dd>
</dl>
<h3>Additional Information:</h3>
<p>Son of Sidney C. and Jessie M. Bowen-Smith, of 2247, Albert St., Regina, Saskatchewan.</p>

which is totally digestible for inclusion in a report. It might be useful to use as a simple method for checking on a suspect record.

~~~~~~~~
Straightforward retrieval of LAC info: if you know the LAC id (in this case 236439) of the soldier use
http://www.collectionscanada.gc.ca/databases/cef/001042-119.01-e.php?&id_nbr=236439

Not-so-straightforward retrieval of attestation record images, creating two links (note that I'm using an actual LAC ID here - in this case 105755):

curl --silent "http://www.collectionscanada.gc.ca/databases/cef/001042-119.01-e.php?&id_nbr=236439" | grep -o 'image_url=.*\.gif' | sed 's/image_url=\(.*\)&.*">.*=\(.*\)/<a href="\1">First page<\/a> - <a href="\2">Second page<\/a>/'

Which returns something like:
First page - Second page

Some files have been digitized, and a PDF can retrieved by running this:

curl --silent "http://www.collectionscanada.gc.ca/databases/cef/001042-119.01-e.php?&id_nbr=236439" | grep -o 'href=.*\.pdf' | sed 's/href="\(.*\)">.*/\1/'

Which returns something like:
http://data2.collectionscanada.gc.ca/cef/9001-10000/9032-35.pdf