HCMC Blogs

October 2, 2020
October 1, 2020

Docker container for link checking

Posted by on 01 Oct 2020 in Announcements

I was able to get the W3C checklink tool working as a Docker container after modifying the Dockerfile found at https://github.com/stupchiy/docker-w3c-checklink/blob/master/Dockerfile to add some missing dependencies; this is my working file:

FROM ubuntu

MAINTAINER Sergiy Tupchiy <stupchiy@gmail.com>

# Install dependencies required for link-checker distribution installation
RUN apt-get update && apt-get install -y \
      cpanminus \
      make \
      build-essential \
      libssl-dev \
      curl \
      libnet-ssleay-perl \
  && rm -rf /var/lib/apt/lists/*

ENV CHECKLINK_VERSION 4_81

ARG CHECKLINK_URL=https://github.com/w3c/link-checker/archive/checklink-${CHECKLINK_VERSION}.tar.gz

RUN set -x \
  && curl -sSL ${CHECKLINK_URL} -o /tmp/link-checker.tar.gz \
  && mkdir -p /usr/src \
  && tar -xzf /tmp/link-checker.tar.gz -C /usr/src \
  && rm /tmp/link-checker.tar.gz \
  && cd /usr/src/link-checker-checklink-${CHECKLINK_VERSION} \
  && cpanm --installdeps . \
  && cpanm IO::Socket::SSL \
  && cpanm LWP::Protocol::https \
  && perl Makefile.PL \
  && make \
  && make test \
  && make install \
  && rm -rf /usr/src/link-checker-checklink-${CHECKLINK_VERSION}

ENTRYPOINT ["/usr/local/bin/checklink"]
CMD ["-h"]

I'm now able to run it using:

docker run -it --rm link-checker checklink -H --recursive https://mapoflondon.uvic.ca > report.html

The plan is to build a complete local copy of e.g. MoEML and then start a Python simple server there, and point the link checker at it.

September 28, 2020

Isolation log week 29

Posted by on 28 Sep 2020 in Activity log

Isolation log week 29:

Work done from home and in the office 2020-09-28 to 2020-10-02

Week 29 total hours: 37.50

G&T hours: +2.5

Monday 2020-09-28

0.25 hours: Update Jenkins servers.
0.25 hours: Admin: Lab scheduling and arrangements for various projects.
1.75 hours: Scancan: Finish encoding of vol 28 article; add fixes per HT; build and publish site.
2.25 hours: Endings: Worked on UBC presentation on staticSearch, and also added staticSearch docs to Endings live site.
0.50 hours: DVPP: Update and test ant tasks to dump dbs on new SQL server; some old code was still around from before the server switchover.
1.50 hours: LEMDO: Triage tickets, and add fixes for two tickets which will be closed if the build succeeds; add descriptive documentation and echo output to many tasks in the build file to make it valid against our documentation schema (finally).
1.00 hours: MoEML: Rewrote home page content, processing and styling so that the news block is no longer drawn automatically from the news pages but is manually editable directly in the info/index.xml file.

Total: 7.50 hours

Tuesday 2020-09-29

0.25 hours: Update Jenkins servers.
0.50 hours: Wendat: Update documentation on installing svn on Macs; contacted JQ to point to the new instructions ahead of our meeting today.
0.50 hours: LEMDO: Noticed that the log parser on Jenkins wasn't flagging a number of things it should be, so updated the log parsing rules.
0.25 hours: MoEML: Fixed JS bug on home page; failed to fix CSS issue on mobile, and passed to PS.
1.25 hours: HCMC: Staff meeting.
0.25 hours: LEMDO: Tweaks to XSLT to fix warnings now shown by Jenkins log parse changes.
1.25 hours: MoEML: Project meeting, and tweaks to schema.
0.25 hours: Wendat: Zoom with JQ to get started with svn and Oxygen on laptop.
0.25 hours: Endings/staticSearch: Fix for missing file issue reported by GL.
1.50 hours: Wendat: Fix for documentation rendering; online training session with ML, BS and EH; created editing cheat-sheet and circulated it.
0.25 hours: DVPP: Pulled latest db data and refreshed TEI metadata for Century Hobby Horse per AC.
1.00 hours: MoEML: Bugfixes, ticket triage, adding new diagnostic for editorial notes lacking @resp, replacing completed diagnostic with Schematron rule.

Total: 7.50 hours

Wednesday 2020-09-30

0.25 hours: Update Jenkins servers.
1.50 hours: LEMDO: More work on Jenkins log parsing; ticket triage; fix for XSLT bug; Project Meeting.
2.50 hours: MoEML: Log parse rule tweaks; new redirect; rewrite of diagnostics rendering for more clarity and convenience.
1.50 hours: MyNDIR: Script for retrieving images from Jonagold, ready for uploading to NFS; training session on svn.
0.75 hours: Endings/staticSearch: Ticket triage; work on presentation for UBC.
1.00 hours: DVPP: Facilitated training session; worked on plan for reader poem view; discussion on stats and plans.

Total: 7.50 hours

Thursday 2020-10-01

0.25 hours: Update Jenkins servers.
0.25 hours: DVPP: Handle two of the RA timesheets.
0.50 hours: HCMC: Discussions around office space, access, and the conference table; moving furniture.
0.50 hours: Maint: More adventures with WSL on my Win 10 vm, trying to get to a point where I can test stuff on WSL. No joy yet.
3.00 hours: Wendat: Analysis of cases of correction and insertion from EP; drafting of new documentation for this with images; updating of documentation CSS and rendering paths to support images and captions.
1.00 hours: DVPP: Rewrite processing pipelines to support quick processing of document subsets by id; write and test build task for preprocessing of poem XML files ready for rendering of reader view of poems.
0.50 hours: MoEML: Switch to blackletter font for pilcrows used as section markers in all documents (Asana ticket).
1.50 hours: Research, test, debug and get working the W3C link-checker docker container, in hopes that it can be used to replace the now-defunct linkchecker software we previously used to check links in project sites. This now works, and I'll be running a complete check on a local copy of MoEML tomorrow.

Total: 7.50 hours

Friday 2020-10-02

0.25 hours: Update Jenkins servers.
1.50 hours: TEI: Meeting with SB to prepare for next week's Stylesheets Working Group meeting, look at ISO 8601 Extensions and TEI implementation, and discuss rendering of egXMLs.
1.50 hours: DVPP: Debug and fix XML pre-processing for poems, and add XSpec tests for key functions.
2.50 hours: MoEML: Build additional process into diagnostics to generate a set of files listing external links that can be checked by a link-checker. Test with W3C checklink tool, refine, and begin fixing broken links and creating tickets for others.
0.50 hours: Keats: Build new release, test, and push to server, per KB.
0.25 hours: DVPP: Refresh TEI from db for Good Words.
1.00 hours: MoEML: More work on fixing broken URLs, refining the diagnostics to catch them, and raising tickets for systematically broken sets.

Total: 7.50 hours

September 21, 2020

Isolation log week 28

Posted by on 21 Sep 2020 in Activity log

Isolation log week 28:

Work done from home and in the office 2020-09-21 to 2020-09-25

Week 28 total hours: 34.00

G&T hours: -1.00

Monday 2020-09-21

0.25 hours: Update Jenkins servers.
0.75 hours: Discussion/documentation of US's EyeVocab algorithm, which is becoming increasingly clear.
1.50 hours: MoEML: With KB, implementation of redirects that were neglected during 6.5 development cycle.
3.50 hours: DVPP: Discussion with team re changes to CSS using pseudo-elements; schema changes to support pc element; re-encoding of all instances where pseudo-elements were used; XSLT and documentation updates to handle these changes; project meeting with AC; first steps in writing rendition transformers, with many use-mention obstacles to overcome.
1.00 hours: MyNDIR: Discussions on contract and training schedule; removal of duplicate file; addition of mouseover showing item id for encoders' convenience.

Total: 7.00 hours

Tuesday 2020-09-22

0.25 hours: Update Jenkins servers.
1.50 hours: MyNDIR: Emails on planning and scheduling of releases and training; implementation of feature request to add mouseover info to images in index listings and gallery.
1.00 hours: HCMC: Staff Zoom meeting.
2.00 hours: MoEML: Project meeting; chasing a couple more missing redirects; ticket work.
0.50 hours: Keats: Troubleshooting MacOS/svn issue for KB. Not yet resolved.
2.00 hours: Wendat: Prep and then online training session, installing svn and Oxygen and basic terminal commands.

Total: 7.25 hours

Wednesday 2020-09-23

1.00 hours: Update Jenkins servers; work through all existing build jobs, cleaning out old build artifacts erroneously archived or left over from aborted builds, to clear some disk space; reconfigure a couple of jobs to reduce cruft.
1.00 hours: MyNDIR: RA orientation to HCMC, intro to project, etc. Added RAs to svn repo.
1.25 hours: LEMDO: Project meeting, and beginning of planning document for possible move from svn to GitHub.
1.00 hours: Wendat: Update documentation; work with JQ in the lab on her first transcription session.
1.50 hours: Scancan: Layout, setup, pre-process and begin encoding new vol 28 article.

Total: 5.35 hours (dental appointment first thing)

Thursday 2020-09-24

0.25 hours: Update Jenkins servers.
0.25 hours: Scancan: fix for missing date in current encoding job.
5.50 hours: DVPP: Implemented the first phase of generating on-the-fly hierarchy-transformed versions of poems so that we can create both normalized and page-based views of each poem for site users (as opposed to editors). This requires first reprocessing the rendition/selector stuff, as well as converting the hierarchy and massaging ids to make sure they're unique. It may make sense to do this as part of generating XML versions of the poems initially, then process those versions, but right now I'm assuming that it's all done in memory as a first stage in creating XHTML5. Also wrestling with unwanted namespace declarations which are making HTML invalid; no solution there yet.
1.00 hours: Wendat: Project meeting.

Total: 7.00 hours

Friday 2020-09-25

0.25 hours: Update Jenkins servers.
1.00 hours: Meeting with SB to plan ticket work for TEI Stylesheets Working Group.
1.50 hours: Wendat: Tweak and run code for adding ids; edit XSLT to make use of the ids, and fix a couple of bugs.
2.00 hours: MyNDIR: Wrestle with the Mac to get a new user set up for the scanning project, and test out / install scanner and graphic editing software.
1.25 hours: Scancan: Finish bibliography for vol 28 article; start encoding main content.
1.00 hours: Meeting with TEH to discuss and develop the tentative proposal for moving to GitHub.

Total: 7.00 hours

September 15, 2020

Upload new files to loi account via personal account on home1t

Posted by on 15 Sep 2020 in Activity log

I uploaded a file using Transmit to the home1t/sarneil folder

cantaloupe.hcmc.uvic.ca% pwd
/home1t/sarneil

cantaloupe.hcmc.uvic.ca% ls -al ToDo.txt
-rw-rw-r-- 1 sarneil tapor 5691 Jun 10 14:52 ToDo.txt

cantaloupe.hcmc.uvic.ca% sudo su - loi
[sudo] password for sarneil:
su: warning: cannot change directory to /home/loi: No such file or directory

-bash-4.1$cp ToDo.txt ../loi/www/fb_dev/

-bash-4.1$ ls -al ../loi/www/fb_dev/ToDo.txt
-rw-rw-r-- 1 loi loi 5691 Sep 15 14:10 ../loi/www/fb_dev/ToDo.txt

To over-simplify a bit
- other than uploading and deleting the files, treat each of two Transmit windows as if they're read-only
- treat the terminal window treat it as if it has write but no read privileges
It's a pain to have to have numerous ftp and terminal windows open to the same directories, but with each logged in as different users (and keep track of which window is which user) in order to see, confirm and do even something as simple as this.

September 14, 2020

Isolation log week 27

Posted by on 14 Sep 2020 in Activity log

Isolation log week 27:
Work done from home 2020-09-14 to 2020-09-18

Week 27 total hours: 35

G & T hours: 0

Monday 2020-09-14

0.25 hours: Update Jenkins servers.
3.25 hours: MoEML: pre-release build, check, test, bugfix.
1.00 hours: MyNDIR: Zoom with PAB to discuss/plan RA training and next release of site.
1.50 hours: Wendat: RA interview, discussion, planning, beginning of conversion of another MS for encoding.
1.00 hours: HCMC/Maint: Discussions on RA workstations and training.

Total: 7.00 hours

Tuesday 2020-09-15

0.25 hours: Update Jenkins servers.
1.00 hours: HCMC: Staff meeting on Zoom.
1.00 hours: Scancan: Corrections to published article and review per author and HT; edit of Garamond No 8 Bold Italic font to add missing character small letter o with ogonek, required for vol 28 article.
5.00 hours: MoEML: Release day for edition 6.5. Lots of proofing, bugfixing, updating, and a project meeting. Final release, then some cleanup of the old content and raising of new bugs related to uncaught errors.

Total: 7.25 hours

Wednesday 2020-09-16

0.25 hours: Update Jenkins servers.
1.50 hours: Wendat: Extract page-image from PDFs; build new Potier Misc MS; prep materials for svn and Oxygen training.
0.25 hours: MoEML: Follow-up on issues arising out of yesterday's release.
0.50 hours: LEMDO: Project meeting.
0.75 hours: Eye Vocab: Data structure mapping based on rough spec from US, with SA. Follow-up questions to US.
0.50 hours: Scancan: More corrections to vol 27 article; published new site build with latest changes.
3.50 hours: Wendat: Prep for training session; training session.

Total 7.25 hours

Thursday 2020-09-17

0.25 hours: Update Jenkins servers.
0.25 hours: DVPP: RA timesheet check and submission.
0.25 hours: Sanitize workstations used in training yesterday.
2.00 hours: Wendat: Add sections to documentation; reorganize documentation; add new page for RAs to site; update schema build process to handle TEI Github branch renaming.
1.00 hours: MoEML: Add new diagnostic to detect ids deleted but not properly retired with a redirect; test and deploy; use modified version to get full details of problem ids from last release for RAs to work on. New ticket for redirect to non-existent id.
1.00 hours: LEMDO: Fix spacing bug and regenerate PDFs; install LaTeX libs on Jenkins to support PDF generation; add Jenkins task to build current PDFs, archive, and clean up afterwards. This saves manually updating GDrive versions every couple of days.
0.25 hours: Maint: Discussion with GN of deployment of new version of MRBS.
1.25 hours: DVPP: Preliminary implementation of function to convert CSS @selector attribute to XSLT @match syntax, along with XSpec tests. Coming along nicely.
0.25 hours: LEMDO: Update publisher info in XML sources.
0.75 hours: Wendat: Project meeting.

Total: 7.25 hours

Friday 2020-09-18

0.25 hours: Update Jenkins servers.
1.00 hours: TEI: Weekly meeting with SB; carry out discussed process for patching the released version of the Stylesheets, and report to Council.
3.00 hours: Wendat: Devised a system for ids for entryFree and form elements in transcribed manuscripts, and described it in detail in documentation, so we can get the team's approval before implementing. Resulting discussion over email, and more edits. Not settled yet. Wrote and tested XSLT to generate/add ids, or to re-id a document should that be necessary.
0.75 hours: GERU: Read and respond to US's responses to our questions. The putative algorithm is becoming slowly clearer, but there are still lots of ambiguities and vaguenesses to tease out and settle.
1.00 hours: DVPP: More work on CSS selector parser, devising approaches to handling pseudo-elements; email to the team for their reactions.
0.25 hours: Endings/staticSearch: Investigation of possible mechanisms for configuring elements to be ignored entirely when indexing; raised enhancement issue for this, which both DVPP and Wendat could definitely use.

Total: 6.25 hours

September 11, 2020