Permalink 09:17:46 am, by mholmes, 64 words, 12 views   English (CA)
Categories: Activity log; Mins. worked: 75

Refined parallel approach

While the massively-parallel approach appeared to be working on my desktop when run overnight, some processes did seem to have crashed near the beginning; on GN's machine the whole thing died within minutes. I've now rewritten it so that only four (configurable param) processes are run in parallel at the same time, to hopefully reduce the load on the machine. We're now testing that.


Permalink 05:20:05 pm, by mholmes, 118 words, 12 views   English (CA)
Categories: Activity log; Mins. worked: 90

Parallel approach to generating match data

Since the existing process was taking a very long time (1652 minutes for 25,600 records), I've devised a revised approach which allows for parallelization of the process. Basically, the driver ant file runs the XSLT with a special parameter that causes the XSLT to write a new temporary ant file as a parallel driver; then the first ant file calls the temporary one, and several processes are kicked off simultaneously. We're both running this overnight to see how fast it goes. There's obviously a lot of tuning we could do in terms of the task division, so we'll definitely be coming back to this, but since the default process will take up to two weeks, cutting it down is essential.


Permalink 04:03:49 pm, by mholmes, 44 words, 14 views   English (CA)
Categories: Activity log; Mins. worked: 240

Back to processing incoming data

With GN, examined our original code for importing the two datasets, and started a revamp/rewrite of it, managed by ant. Currently running a full similarity metric test against the latest CGWP version. Will take days. May be able to split and parallelize it.


Permalink 03:42:14 pm, by Jas, 43 words, 59 views   English (CA)
Categories: Activity log; Mins. worked: 90

April 3rd

Still working through the Ontario locations.
Today i found a person with two entries with different LACID's.
Joseph Harold Code (pid:834415) - LACID 7900 (which is actually ANDERSON, ARNOLD ALBERT's LACID)
Joseph Harold Code (pid:934457) - second entry has the LACID 107900 which is correct.


Permalink 02:42:55 pm, by Jas, 15 words, 56 views   English (CA)
Categories: Activity log; Mins. worked: 360

March 11 - 15

James Albert Thompson seems to have two entries.
Have been working on Ontario. Down to ~3,100.


Permalink 04:29:22 pm, by Jas, 37 words, 63 views   English (CA)
Categories: Activity log; Mins. worked: 450

Feb 26 - 28

Manitoba is pretty much finished, with 23 entries remaining.

Made a mistake: The match between Victoria Man. and Holland river Ont. is a mistake and should be removed.
The Appropriate match is with the Victoria Rural Municipality Man.

Permalink 01:26:29 pm, by AJ, 8 words, 67 views   English (CA)
Categories: Activity log; Mins. worked: 60

February 28th 2018

Worked on Quebec, down to 727 places to match


Permalink 04:29:56 pm, by AJ, 143 words, 67 views   English (CA)
Categories: Activity log; Mins. worked: 240

27 February 2018

-Continued work on Quebec, on the 'M's, 773 left. - Wilfred Meagher file has incorrect province for Glengarry, should be Ontario. It is corrected on the document but not CGWP. - Private John Belt file says Granville, QU should be transcribed as, Georgeville, QU. - Private Frederick Emerson Sunstrum file incorrectly transcribed as Guyon,QU. Should be Quyon, QU. - Private Omer Sevigny file, POB,should be transcribed as Ham-Nord QU, Not Hemmond?, QU. - Léger Turcotte file says birthplace is Jeune Lorette, QU. Should be transcribed as Loretteville, QU. - Private Ernest David P.O.B should be transcribed as Joseph Farm, Maniwaki, QU. Rather than Joeseph, QU. - Oliva Lanouette file POB should be transcribed Sainte-Anne-de-la-Pérade, QU. Rather than La Prade, Champlain County, Quebec. - Private Ernest Tremblay file POB should be transcribed as, Lac-Cayamant, Quebec. Rather than Longeault, Quebec


Permalink 03:42:14 pm, by AJ, 39 words, 69 views   English (CA)
Categories: Activity log; Mins. worked: 135

26 February 2018

Worked on Quebec Locations up to the end of 'F' with 12 unknowns. John Angus McDonald (Quebec Born, Died 1916) has one page of records that belong to a different John Angus McDonald who lived through the war. (End of file)


:: Next Page >>


Development blog for next version of the Canadian Great War Project. The production site is at https://cgwp.uvic.ca/, and the development site is at https://cgwpdev.uvic.ca/ (access controlled)

Some sections of the development site are not explicitly linked via menus. See the post titled "CGWP development URLs" for details.


XML Feeds