Archives for: 2012


Permalink 03:31:14 pm, by sarneil, 334 words, 165 views   English (CA)
Categories: Activity log; Mins. worked: 90

proceeding with building documentation table

Got answers to questions and data spreadsheet from PD.
The production front end is pointing at the vihdev db. I tried to back that up using the psql command provided by Jaime and I got a file of zero bytes. I then tried to use the export feature in phypgmyadmin (or whatever it's called) and that brought the db service down. Here's the error message, which suggests that the device is full, but we now think is actually a permissions problem:
SQL error:

PANIC: could not write to file "pg_xlog/xlogtemp.24088": No space left
on device
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Sys-admins cleaned things up and restarted the service within a few hours. Thanks.

I tried connecting to the vihistory db running in the same environment, but the connection failed.

So, until I get a successful backup of the data in the server, I'm very hesitant to do any work on it. Here's what I've proposed to the sys-admins:

1) export and clear big log table from db
- write table search_log in db vihdev to file
- clear contents from table search_log in db vihdev
NB When I recently tried it from my Mac, I got a search_log.err file of
size 0 bytes. Since the last time we successfully executed the command
below the db engine has moved to mango.
psql -W -U viAdmin -h -A vihdev --command "copy
search_log to stdout;" > search_log.err
NB table has 6.5 million records, each about 1K = 6 gigs

2) create a new db to use as the production instance
- clone table vihdev, name clone vihprod
- drop db vihistory
- modify acl on vihprod based on settings for vihdev
NB I have to change connection strings on site

3) set up long-term plan for write-then-clear search_log table
NB cron job vs manual
NB frequency

4) ensure I can migrate future changes in vihdev db to vihprod db


Permalink 03:47:20 pm, by sarneil, 432 words, 216 views   English (CA)
Categories: Activity log; Mins. worked: 90

hist : cha : adding banner image

PB sent me a large jpg to use as a banner on the conference site. Took a couple of guesses to get something the correct dimensions. Also modified the bg of the image from white to the taupe colour used on the site - they were OK with that.

Turned out that
- headerTitle div had a number of peculiar styling rules
- #header h1 had odd padding settings which resulted in the banner being noticeably off-center
I overruled by adding a selector in the cha2013styles.css file. Noticed that a lot of elements have a lot of instances of selectors and conflicting style rules applied to them - sometimes 2 or 3 selectors for the same id in the same css file.

Finally, it seems that the CMS doesn't like comments in css files, so I can't include any.

Here's the contents of the modified css file to date (with comments added here by me):

/*hide certain items in the nav-bar at the top of the page*/
not logged in
li:nth-child(1) = home
li:nth-child(2) = about
li:nth-child(3) = log in
li:nth-child(4) = account
li:nth-child(5) = search
li:nth-child(6) = announcements
logged in
li:nth-child(1) = home
li:nth-child(2) = about
li:nth-child(3) = user home
li:nth-child(4) = search
li:nth-child(5) = announcements

div#navbar li:nth-child(1),
div#navbar li:nth-child(2),
div#navbar li:nth-child(4) {

div#navbar li a[href$=account] {
div#navbar li a[href$=search] {
div#navbar li a[href$=announcement] {

/* make header have thinner border on bottom and mirror it on top */
#header {
border-bottom: 1px solid #898E79;
border-top: 1px solid #898E79;

/* get rid of odd padding around h1 in header to ensure contents are centered */
#header h1 {
padding: 0 0 0 0;

/* headerTitle has a barrage of selectors and style rules, over-rule them all
most importantly over-rule the fixed height so the banner image can be contained elegantly */
#headerTitle {
height: 100%;
padding-bottom: 0px;
padding-top: 0px;
margin-bottom: 0px;

Permalink 03:30:27 pm, by sarneil, 164 words, 174 views   English (CA)
Categories: Activity log; Mins. worked: 360

1891 check census draft added to dev site

At request of PD, I processed the the raw checkcensus csv files he submitted into something suitable for upload to database. Oddities in data that had to be resolved: street numbers such as 7-9 or 12 and a half, number of occupants such as 20+, 20plus, 20 to 30 etc., quotation marks and commas in the dataset being confused for quotation marks and commas as delimiters.

Also set up the table, created the fields including an entry in the sequences table, all based on similar work done for the 1871 census, which was also a kind of one-off data set.

Did some twiddling with the presentation of results (particularly with how to represent the various note fields). For those records that have an id for the 1891 Dominion census, wrote code to generate a link to the full record for that individual in the Dominion Census, but have not modified the 1891 Dominion census in any way (e.g. to include a link back to an entry in the 1891 Check census).


Permalink 04:38:11 pm, by sarneil, 133 words, 120 views   English (CA)
Categories: Activity log; Mins. worked: 60

issues with updating records to census_1911 table

Looked into what's involved in updating 1400+ records in the census_1911 table.
Have spreadsheet from PD. About 37 of the fields in the db can be just read in from the spreadsheet. About 12 of the fields in the db have to be calculated based on values in the spreadsheet (the nature of the calculation varies for each of the 12 instances. The notes field in the PG table describe the needed calculations. Not sure how Jamie did this last year, but it looks like I'll have to write a bunch of code to do that processing and generate the csv needed.

Then I'll need to tread carefully regarding the actual upload. I think I'll need to delete the existing records and then upload (using the copy command, or possibly the import feature in the admin client).

Permalink 04:20:27 pm, by sarneil, 102 words, 145 views   English (CA)
Categories: Activity log; Mins. worked: 120

create street renaming and address renumbering pages

PD asked me to create one page to display a table of all the streets (or portions of streets) that have been renamed over the time range of the VIHistory datasets. I did that by running the tab-delimited through a few regexs to produce a styled html table and put that into a new page.
PD also asked for a similar page to display a table of all the addresses that were renumbered in 1907. That took a bit more regex processing, as I wanted to created a heading for each street and then a table for all the addresses on that street.

Permalink 04:13:05 pm, by sarneil, 46 words, 167 views   English (CA)
Categories: Activity log; Mins. worked: 60

minor changes to appearance of links page

Made some text corrections and updated some URLs on the links and maps landing pages. Also added one occupation to the occupations table (but not the occupations_cnv table) in both the vihistory and vihdev instances (though the latter is I think no longer in use).


Permalink 04:11:41 pm, by sarneil, 38 words, 165 views   English (CA)
Categories: Activity log; Mins. worked: 60

only basic search for building permits

Modified global variables so only the basic search is available to user. Also added a help file specifically for the business permits, changed global variable that writes link to it and edited the content of the help file.


Permalink 04:30:26 pm, by sarneil, 229 words, 206 views   English (CA)
Categories: Activity log; Mins. worked: 360

create building permits search interface

Made copies of a number of ta (tax assessment) files, renamed them and edited them to work with the building_permits table in the db. Did this in the dev instance in my account on our server.
Files I've added:

The link to the searchbp.php page is on the tax assessment page (ta/taxassessment.php).

Emailed JL and PD for guidance on which fields to include in search interface(s) and which results fields are primary and which secondary.

More headaches with svn than the actual code (as usual). I had in the repo and on my local drive a folder (bp) containing 1 file. I did an svn delete and the file deleted on the local instance, but not the folder. When I then did a commit, I got a file out of date error. When I tried various ways of sorting out this problem I ended up consistently geeting "/path/on/local/drive/' remains in conflict" errors.
Googled it and discovered I'm not alone. Quite a number of people renamaing, moving or deleting files that have everything going smoothly except for one file or folder that some gets into "conflict"
svn resolved path/to/conflicted folder
svn update path/to/conflicted folder
svn commit -m "resolving conflicted folder or whatever"


Permalink 03:38:57 pm, by sarneil, 174 words, 152 views   English (CA)
Categories: Activity log; Mins. worked: 360

add building permits table and populate it

Created a building permits table in the vihdev db. Noticed that to auto-increment the building_permit_id field, you have to reference a sequence, so created the necessary sequence modelled on others I found in the db (census tables).

Processed the raw data file (spreadsheet) into normalized data (typed a couple of the data fields that I could e.g. int or date and normalized data to comply with the constraints I had established e.g. length of varchar fields). Saved that as a CSV (rather than tab-delimited) as the documentation seemed to favour the CSV approach.

Only substantial fiddling I had to do with the data was for all the records whose date field was only a year (e.g. 1889), I arbitrarily assigned them the 1st of January (e.g. 18890101) as the date field requires 8 digits.

Once that instance uploaded successfully, did the exact same thing in the production instance, just so I have a second copy of the thing somewhere.

Once that was all working in the dev instance of the db,


Permalink 01:49:32 pm, by sarneil, 57 words, 168 views   English (CA)
Categories: Activity log; Mins. worked: 180

revised branches with updated imap stuff

Replaced the contents of the trunk, Alex branch and the backup branch with the files as updated by Martin and Greg, so they're up to date.
Gave AD all the svn, web account and db connection info he should need to get to the files, check them out, and post them to the web space for testing.


Permalink 12:04:59 pm, by sarneil, 79 words, 206 views   English (CA)
Categories: Activity log; Mins. worked: 15

add new tag or branch to svn repo

note to self on nuts and bolts
on local file system:
create the folder structure you want (if you're copying an existing local instance of an svn project, you have to delete the .svn file from each folder in that project)
on command line,
cd to parent folder of the one you want to add (that parent folder has to already be in svn)
svn commit -m "message about adding new folder"

Permalink 10:46:15 am, by sarneil, 118 words, 90 views   English (CA)
Categories: Activity log; Mins. worked: 15

editing config files in new svn branches

There are three files in the site which contain database connection strings:

In each of these three files, the values for the database connection string have been replaced with placeholders. You have to make a copy of each of those files with the following names:
In the copies, substitute the correct values for your connection string.

If the folder is in svn (which it probably is), you'll need to use svn add to add each of the files to the repo, then do your svn commit.


Permalink 02:05:58 pm, by mholmes, 76 words, 132 views   English (CA)
Categories: Activity log; Mins. worked: 60

Streetcar map fixed; other issues o/s

I've now fixed the streetcar map problem, by changing all filenames and references so that they're consistently referring to 1939 instead of 1936. The fixes have been committed to trunk, and put up on the website.

The other two issues remain outstanding; PD will get back to me with the correct firemap URL at Malaspina, and we'll wait until the DNS has been changed before addressing the problem with Firefox and captcha, since it seems to be cookie-related.

Permalink 08:43:47 am, by mholmes, 292 words, 90 views   English (CA)
Categories: Activity log; Mins. worked: 45

PD has checked the site: a couple of remaining issue

PD has checked the new VIH site on taprlans/www, and reports only these issues:

  • Captcha reports errors for him on FF. I can't reproduce this -- it works fine for me -- so waiting for more details.
  • There's a link on the maps.php page pointing to the "1936 streetcar map" which fails, defaulting to the panorama, because it should point to 1939. The confusion arises thus:
    The map itself has the date "1939" on it.
    The .map file (the basic definition file) is called
    However. it contains pointers to images called:
    and this page:
    has 1936 in its caption. However. if you go to the map viewer,
    click on the Maps menu and drill down to it. you see the caption "1939
    - Streetcar routes".
    I think that:
    	- The actual date is 1939.
    	- The images are wrongly named, as are the pointers to them in
    the .map file.
    	- The caption on the maps.php page is wrong. whereas the Maps 
    menu in the map viewer is right.
    Waiting for PD to confirm my analysis before changing the image file names, the .map file, and the caption/link on maps.php.
  • PD reports this:
    The link on the page describing the 1885 Fire Insurance Plans of Victoria 
    needs to be changed. The page in question is located at:
    The hyperlink should be redirected to:
    [The link currently points to an obsolete server -
    However, the new URL is about 1891 panorama images; it has nothing about the 1985 Fire Insurance Plans. Waiting for the correct URL from PD.


Permalink 10:39:32 am, by mholmes, 104 words, 99 views   English (CA)
Categories: Activity log; Mins. worked: 30

Removed a browser restriction on the VicFire 1891 map

There's one special map that uses SVG for its interface, and which was originally working only on FF; DB had put a complete block on other browsers, but now most of them support SVG so I've removed that block. Browsers that don't support SVG should get with the program. Committed that change to SVN, but I was unable to commit the bulk of my additions (code which was not originally in SVN, but should have been) because we ran out of disk space on We'll be switching to the new SVN on Monday, so hopefully this problem will be solved.


Permalink 03:33:08 pm, by mholmes, 187 words, 101 views   English (CA)
Categories: Activity log; Mins. worked: 360

Got the maps working

Got the site basically working by doing this:

  • Moved everything from taprhist/vihdev/www to taprhist/www (it really doesn't like living in that odd location, and there are hard-coded paths in several text files).
  • Removed "vihdev" from paths in config files.
  • Cleaned out the /home1t/taprhist/www/content/maps/cache folder so it had to start rebuilding.
  • Now we found that most maps were working, but a handful were failing. The failing maps had one line in their .map file:
          "queryable" "true"
          "tile_source" "cache"     <-- This line has to be removed.
    Our surmise is that this problem line causes the server to construct a broken path to a cache folder that doesn't exist or isn't writable, and therefore it cannot read or construct tiles.

We now propose to have the DNS repointed so that points at taprhist/www, and keep the live site there.

Greg also noticed that is broken; outside of the ring it's pointing at DNS servers, so he's emailed PD to get him to fix that on the domain host.


Permalink 03:52:25 pm, by sarneil, 417 words, 108 views   English (CA)
Categories: Activity log; Mins. worked: 300

missing maps resist debugging

I've been working for the better part of the last couple of days trying to figure out why the imap maps in ViHistory are not appearing.
Problem appears with the production front end and a test front end connected to either the old db server or the new db server.
Problem appears with a test front end which is the production front end minus the captcha code (which is the only code that has changed since Jamie left us a working site).
We weren't getting errors when we trolled the server logs on lettuce.
We did get the following errors from the sysadmin:
Apache error log:

[Tue Mar 06 09:16:21 2012] [error] [client] [Tue Mar 6
09:16:21 2012].616424 loadSymbolSet(): Unable to access file.
(/home1t/taprhist/www/content/maps/user/symbol/generic.sym), referer:

AND syslog:

2012-03-06T09:16:21-08:00 user.notice
php-cgi: PHP Warning: [MapServer Error]: loadSymbolSet(): (/home1t/taprhist/www/content/maps/user/symbol/generic.sym)
2012-03-06T09:16:21-08:00 user.notice in: /home1t/taprhist/vihdev/www/content/maps/htdocs/init.php on line 125
2012-03-06T09:16:21-08:00 user.notice
php-cgi: PHP Warning: Failed to open map file /home1t/taprhist/vihdev/www/content/maps/user/map/ in /home1t/taprhist/vihdev/www/content/maps/htdocs/init.php on line 125
2012-03-06T09:16:21-08:00 user.notice
php-cgi: PHP Fatal error: Call to a member function getMetaData() on a non-object in /home1t/taprhist/vihdev/www/content/maps/htdocs/init.php on line 131
2012-03-06T09:16:21-08:00 local0.debug
suphp_wrapper: 0 PHP5

That init file is as provided by the imap people, so I really, really doubt it is causing the problem, though it is the where a problem occurs. It looks like that file is trying to create objects based on what it reads from some kind of config file, and somehow that process is breaking down, so the object doesn't get created, so invoking a method on the (non-existant or empty) object throws the error. MOre precisely, it looks like the config.php file is supposed to create an array in the variable aszMapFiles and then those values are used in the init file, but for some reason something is failing with the way that array and associated variables are being populated.


Permalink 04:57:58 pm, by sarneil, 51 words, 68 views   English (CA)
Categories: Activity log; Mins. worked: 180

post-gres based maps not showing up

User reported that none of the post-gres based maps (insurance etc) for ViHistory are appearing. The imap infrastructure appears in the browser window, but no image tiles. PD forwarded this and I confirmed it. Spent a number of hours with Greg and Drew trying to solve the problem. Still not resolved.


Permalink 04:00:49 pm, by sarneil, 158 words, 85 views   English (CA)
Categories: Activity log; Mins. worked: 60

db on new db server working

Drew migrated the viHistory and vihdev postgres db's to the new db server (
I created an instance of the viHistory site at and changed the connection string to point at the postgres service on the new db server ( rather than the old db server ( I didn't change anything else. After a couple of iterations through errors coming back from the db, Drew made the necessary adjustments and the connection is now working to the new db.

What I've got to do now is take the vihdev db on the server and put it into the viHistory db on the server. Then when we change the connection string for the production site, we change the name of the db it's pointing to as well as the server it's pointing to.


Permalink 02:48:22 pm, by sarneil, 261 words, 72 views   English (CA)
Categories: Activity log; Mins. worked: 180

create tags and branch for Alex

The trunk I started with has the captcha modifications in it.
I created a tag called beforeAlex which is a copy of the trunk corresponding to the publicly accessible site.
I then svn rename the file to and abstracted the connection string values and committed that. I searched all the files for any other instance of the connection string values and did not find any.
I then created a tag called configAbstracted which is a copy of the trunk with the config file modified (that's the only difference between the configAbstracted tag and the beforeAlex tag).
I then created a branch called alex which is identical to the configAbstracted tag (at least until Alex starts modifying it).

Still to do:
- create an svn user for Alex with rw privileges on
- create a user in the database for Alex with read privileges, which he can use to modify the config_Edit_Me.php file.

Had a puzzling time with this: file originally called was renamed to and file originally called was renamed to, but I forgot to do svn rename for those. Subsequently had a miserable time trying to commit to the repository as I kept getting File " is out of date, path not found", but only for that file. I eventually svn delete 'd the entire svnrepo folder on my local computer and checked out the entire repository and the errors went away.


Permalink 12:30:38 pm, by sarneil, 790 words, 107 views   English (CA)
Categories: Activity log; Mins. worked: 240

how to implement captcha in vihistory searches

To make captcha work, you need to make modifications to the included form file and to the file that invokes the form.
// I'm using the $_SESSION['captchaPassed'] variable, so that even if the user moves from form to form or query to query, they only have to verify one captcha per session
// if $_SESSION['captchaPassed'] is not set, the user has not yet tried to validate a captcha field in this session
// if $_SESSION['captchaPassed'] is set and has value false, the user has tried to validate a captcha field in this session and failed
// if $_SESSION['captchaPassed'] is set and has value true, the user has successfully validated a captcha field in this session

I'll need to rewrite the code in the annotations section to call on the instance of captcha in the captcha folder rather than the one in the annotate folder.

Because the js file currently assumes the call is coming from a file at a certain level in the site hierarchy, and the search and annotate files are at different levels in that hierarchy, I'll have to either write a separate function for the annotate call, or modify the calls to the loadcaptcha function so they pass in the path from themselves to the root as a second argument.

Changes to the actual search form (e.g. censusformbasic) as seen in censusformbasicCaptcha.php

1) Make sure this line points to correct file (for production, should be searchcensus.php) :

$search_arg = "searchcensusCaptcha.php?start=0&form=$form&show=n";

2) Add this block at end of form:

if ((isset($_SESSION['captchaPassed'])) && ($_SESSION['captchaPassed'])) {
//do nothing
} else {
<fieldset class="">
To verify that you are a real, live human and not a spambot,<br />
enter the text you see in the image below.<br />
If you can't read the text, click
<a href="javascript:loadcaptcha('captcha');"><b>here</b></a>,
or see the <a href="<?php print $doc_root; ?>/help/annotation.php#verify">help page</a>.
if ((isset($_SESSION['captchaPassed'])) && (!($_SESSION['captchaPassed']))) {
print '<br /><br /><span style="color: #ff0000; font-weight: bold;">The text you entered does not match the image. Try again. </span>';
<br />
<label for="verifier">Image text (5 letters)</label>
<input type="text" name="verifier" id="verifier" value="" />
<img id="captcha" src="<?php print $doc_root; ?>/captcha/captcha.php" width="200" height="60" alt="Verify" style="border: 1px solid #000000;;"/>
} // end of if isset($_SESSION['captchaPassed']) ...

Changes to the page calling the form (e.g. searchcensus.php) as seen in searchcensusCaptcha.php.

1) Add these blocks to head:

* Load captcha library.

<script type="text/javascript" src="<? print $doc_root ?>/script/loadcaptcha.js"></script>

2) Make sure this line points to right file (for production, should be searchcensus.php):

$search_page = "searchcensusCaptcha.php";

3) Add this block immediately after the first "if (isset($_GET['start'])) {"

//SA we need to defy the spambots
// I'm using the $_SESSION['captchaPassed'] variable
// if $_SESSION['captchaPassed'] is not set, the user has not yet tried to validate a captcha field in this session
// if $_SESSION['captchaPassed'] is set and has value false, the user has tried to validate a captcha field in this session and failed
// if $_SESSION['captchaPassed'] is set and has value true, the user has successfully validated a captcha field in this session
// need to do the CAPTCHA test before any query gets sent to the DB
// Only do the test if the user has not tried to validate yet, or if the user has tried to validate and failed
// my actual is test is : if (NOT ((session var is set) AND (session var is true)) )
if (!((isset($_SESSION['captchaPassed'])) && ($_SESSION['captchaPassed'] == true))) {
if (PhpCaptcha::Validate($_POST['verifier'])) {
$_SESSION['captchaPassed'] = true;
} else {
$_SESSION['captchaPassed'] = false;
if ((isset($_SESSION['captchaPassed'])) && ($_SESSION['captchaPassed'] == true)) {
// We know user has successfully validated the captcha, so do the query and display the results

4) add this block immediately before "} // end of if isset($_GET['start'])"

} // end of if isset ($_SESSION['captchaPassed']) ...

5) make sure this line points to correct file (that is, censusformbasic):

require("$doc_root/inc/"); //SA was

6) add this block immediately before the second "if (isset($_GET['start'])) {"

if ((isset($_SESSION['captchaPassed'])) && ($_SESSION['captchaPassed'] == true)) {

7) add this block after "pg_free_result($result); }}"

} // end of if isset($_SESSION['captchaPassed']) ...


Permalink 04:29:51 pm, by sarneil, 118 words, 97 views   English (CA)
Categories: Activity log; Mins. worked: 60

set up svn, reconcile various copies of vihistory site

Checked out the vihistory svn repository at into a directory on my mac
Copied the files down from the production site (taprhist/vihdev). Next I need to compare the two to make sure that the repository and production site are in sync.

Once that's done, then I can start work on implementing the captcha code from the annotations page into other forms, name the searchcensus one.

Also, the production site has .svn folders in it, which might be the result of using an svn client on that set of folders, or may just be an artefact of ftping files up to the server from another location that was in svn.


Permalink 04:34:40 pm, by sarneil, 129 words, 77 views   English (CA)
Categories: Activity log; Mins. worked: 120

looking into Captcha

Looking into captcha for VIHistory. One simple-looking candidate is Securimage, so I might try that. It's pure php (no js). I have downloaded it and created a simple test file which works fine in the VIHistory account. Hooking that into DB's code for form handling will not be too difficult hopefully. Martin and Greg reported that there is an existing captcha thingee on the annotations form (which I confirmed) but that it was JS dependent, so could be got around by simply turning javascript off (which I did not confirm). I'll have to get more details from them. Right now I don't see what securimage will get us that the current captcha thingee doesn't, as the existing thingee seems to work just fine with javascript turned off for me.
Permalink 04:28:09 pm, by sarneil, 124 words, 80 views   English (CA)
Categories: Activity log; Mins. worked: 30

set up new student

CC has new student ES working FrancoToile. First task is copy editing.
Got her to log in, then I created an SVN collection on Pomme.
She edited one file, then committed the collection, which worked.
She then tried updating, but of course there were no changes to any of the files, so the client reported that the directory had been skipped - not sure if that is the correct behaviour or not. I'll have to change one of the files as me before she comes in tomorrow and see what happens when she then logs in and updates.

Still have not looked at the problem with the rng file not allowing attributes and elements that it should be, but the schema is being invoked.


viHistory is a web site that is a teaching, learning and research tool. It's principally about the history of Vancouver Island in British Columbia, but it is also a vehicle for exploring the larger field of Canadian history during the late 19th and early part of the 20th century. It allows census, directory and tax assessment roll data from the late 19th and early 20th centuries to be searched in many ways. It also incorporates IMaP to display historical maps. The project director is Dr. Patrick A. Dunae.



 << Current>>
Jan Feb Mar Apr
May Jun Jul Aug
Sep Oct Nov Dec

XML Feeds