Carrier dictionary conversion
Posted by mholmes on 26 Jan 2011 in Activity log
On ECH's behalf, I'm helping to convert a Carrier dictionary to a format that the FirstVoices folks can use. I received the dictionary today in Access format, and was able to take the following steps:
- Used MDB Viewer to open the db file, and exported the schema, as well as the contents of the two tables inside it. The tables are "Central Carrier Dictionary" and "export_file". The former has 3079 entries, and the latter 2213. They have the same fields in them, but in different orders. The field list in the former looks like it's in the wrong order, whereas the latter is in a more logical sequence. It looks as though I should be working with the data from the first table, but aiming at the order of the second, but I'm still waiting for confirmation from the authors. In any case, for FV purposes, the order of these fields is not really important.
- Took the exported CSV files and used OpenOffice Base to open them as a database. This seemed to work fine. I was then able to select each table, right-click it, and Copy, then Paste into a spreadsheet in OpenOffice Calc (opening the CSV files directly in Calc gave me an error suggesting that the maximum row count had been exceeded).
- From Calc, I was able to save as a FODS file, and then I opened the FODS files in oXygen.
- I've written two stages of transformation starting from the FODS file. The first "cleans up" the FODS by removing a bunch of fields we don't need, and by expanding all instances of empty table cells with
@table:number-columns-repeated(a colspan setting). This gives a cleaner matrix to work with. - The second transformation is specific to the first of the two tables in the db. It converts the modified FODS file into a customized XML structure which is simple, and has descriptive tags instead of the spreadsheet tags. It's specific to the one file because it names the columns with tags based on the column headers; since the columns are in a different order in the second table, it wouldn't work correctly without re-ordering.
The next stage will be to write the final transformation to produce whatever the FirstVoices folks need. Since I don't yet know what that is, I can't start on it, but I can start writing the Java collation class that will do the sorting. That's my next task.