CGWP

We've been using the slightly modified HISCO codes from VI History, which included levels of specificity that do not exist in the published version of HISCO.

For example, we have
1) a bunch of Managers (Cement Works, Gas Works, Lime Kiln, Mine, Powder Works, Sawmill, and Tanning Company) that *could* all be placed in a more inclusive category, like 22620 "Supervisor and General Foreman (Mining, Quarrying and Well-Drilling)"

2) 50ish posties (postman, postmaster, assistant postmaster, expressman, post office assistant) that could possibly be placed in the Mail Distribution Clerks category (37000-37090) but are listed as 35210, 35211, and 35910.

3) 8 store managers that are listed as hisco 40020,40030,40032, and 40033 but could be handled by the 21000-21990 range

4) auto salesmen/dealers that are listed as 48000, ticket sellers listed as 49090 - TO DO: what to list them as?

5) steamship pursers listed as 50040 - TO DO: what to list them as?

6) housekeeper, stewards, etc listed as 52020,52032, and 52040 - TO DO: what to list them as?

7) gentleman farmer, farm manager, and gamekeeper listed as 60040, 60030, and 60020 - TO DO: what to list them as?

8) hunters listed as 65000 - TO DO: what to list them as?

9) foreman of various descriptions listed as 70000 - 70150 - TO DO: what to list them as?

10) 1994 funky ones (X0000, X0010, X0015, X2000, X2280, X3010, Y1100, Y1105, Y1123, Y1125, Y1200) that correspond to mostly soldiers, gentlemen, cutters, and assorted oddballs (illegible, trailing ?) - TO DO: what to list them as?

Importing the CGWP records that have matching CVWM rows produces about 10,600 results.
Of those, 173 seem to be duplicates.
We need someone to go through them and weed out dupes.
This is the sql to get those records:

select * from transfer ou
where (select count(*) from transfer inr
where inr.cwgc_id = ou.cwgc_id) > 1

1. Incorporate other data - first pass done (added 7800)
2. Incorporate images in to main soldier display - done
3. Incorporate awards in to main soldier display - done
4. Add ranks-held listing in details page - done
5. Drill-down searches for POE, POB, Cemetery, Awards,Memorials, Regiment/Battalion, etc.
6. Prose for main page, about
7. Placeholder text for the rest
8. Search help for search page
9. Future plans/features for search page, war diaries page, stats page, etc. (some of the static content can be put here, like the stats on http://canadiangreatwarproject.com/writing/cefStats.asp)
10. New top level page(?) for 'essays' - for e.g. the 'Canada in the War' section of existing site
11. New top level page for 'Collections' - to contain 'From the Front' materials
12. Widgets: famous Canadians, soldiers died on this date/today in history
13. Convert date_of_death to date field - done

The first-pass of the XSL merge looks like it was extremely successful, and I think I understand the results well enough to understand what the next (last?) pass needs to produce.

In its current state, the new person table appears to exclude approximately 20,000 records from the original CGWP dataset. This number includes duplicates, and records excluded due to a complete lack of viable data (e.g. Harry Johnston, cgwp_id 30 - we only have a name, and the notes say 'Unknown Likely WW1') as well as records that had no matches in the LAC data.

The first two sets of exclusions (dupes and inadequate data) make sense to leave out, but the others need to be addressed. This will include Canadians that signed up with foreign forces, Merchant Marine, etc., as well as records that came from CVWM, CWGC.

We need to normalize a bunch of stuff from the original dataset: occupations, country names, city names, regional names, cemeteries, and religions.
It makes sense to add these to the AdaptiveDB and have ST do the normalizing.