Maintenance

I edited my cdmirror script to accommodate building a PXE-boot bundle using a configurable target distro (it used to only build the current LTS, but now I can set it to build any distro) so I can now do tests on PXE builds of the soon-to-be-released 20.04.

It looks like the apt-mirror script project is dead. The repo directory structure has changed since 2018 and when I added 20.04 to the mirror it did not download a bunch of stuff.

So, I'm testing debmirror. Which also has problems, but is still being developed. The version of Debian I'm using on guava uses an old version of debmirror, so I grabbed the script off of the git repo and installed it locally.

Along with a wrapper script it seems to be much easier to use than apt-mirror was. Fingers crossed it works well when I start to run it as a cron job.

With the new mirror I've had an initial success with a 20.04 PXE build of one of our new lab NUCs.

It's really 1 command - 2 if you don't already have a key.
If you *don't* have a key already, create one with:
ssh-keygen -t rsa -f ~/.ssh/id_rsa -q -N ""
This (quietly -q) creates an RSA-2048 key in the usual place with NO passphrase on the key (-N "" provides an empty string for the new passphrase).

Once you have a key, you only need to run a command like this to push it to a remote machine:
cat ~/.ssh/id_rsa.pub | ssh ${REMOTE_USER}@${REMOTE_HOST}
where ${REMOTE_USER} is your login name and ${REMOTE_HOST} is a machine name or IP address of the remote machine (like myserver.uvic.ca).

I wrote a slightly more interactive script that does this for you. It's in the Utils repo as well, but I've attached a snapshot version to this post.

Static sites often require a CORS- and AJAX-supporting server to function properly, so it's helpful to run one locally. This is the script I've put together for doing that:

# This was created based on info from various online instructions sites.
# First you need to create a cert so that SSL will work.
# Generate server.pem with the following command:
#    openssl req -new -x509 -keyout server.pem -out server.pem -days 365 -nodes
# run as follows:
#    python pyserve.py
# in the folder where the web materials are.
# Then in your browser, visit:
#    https://localhost:4443

import http.server
import ssl

# Not that we're allowing CORS here, because we need to run local sites which 
# use remote resources.
class CORSRequestHandler(http.server.SimpleHTTPRequestHandler):
    def end_headers(self):
        self.send_header('Access-Control-Allow-Origin', '*')
        self.send_header('Access-Control-Allow-Methods', 'GET')
        self.send_header('Cache-Control', 'no-store, no-cache, must-revalidate')
        return super(CORSRequestHandler, self).end_headers()

# If you don't need remote resources, just use this:
#httpd = http.server.HTTPServer(('localhost', 4443), http.server.SimpleHTTPRequestHandler)
httpd = http.server.HTTPServer(('localhost', 4443), CORSRequestHandler)
httpd.socket = ssl.wrap_socket (httpd.socket, certfile='/home/mholmes/WorkData/scripts/pyserve/server.pem', server_side=True)
httpd.serve_forever()

I usually add an alias link in my .bashrc file like this:

alias pyserve='python ~/WorkData/scripts/pyserve/pyserve.py'

so I can just run "pyserve" in the folder where the static site is.

The new machines (NUC8) support WOL but only in S3 or S4 sleep. Right now I've got suspend/hibernate disabled so WOL won't work on the new machines. This is on the todo list.

For the last couple of weeks I've been noticing some very weird behaviour in the two Jenkins build jobs where the svn repo has GitHub externals. The projects simply build all the time, whether or not there's been any change to any of the repos (svn or git).

We can't have that, of course; it's only happening to two repos right now, but it as more mini sub-projects get shared between repos, we'd end up with Jenkins simply queuing builds endlessly.

I'm still researching this, but I suspect it might have something to do with branch information; when (say) the DVPP repo is subscribed to the master branch of the staticSearch git repo, the master branch may not change, but the dev branch might; that means you end up with weird svn info like this (running svn info . in the external directory inside the Keats project):

URL: https://github.com/projectEndings/staticSearch.git/branches/master
Relative URL: ^/branches/master
Repository Root: https://github.com/projectEndings/staticSearch.git
Repository UUID: 04f9d699-67a6-817a-6227-ce57b7db7bff
Revision: 588
Node Kind: directory
Schedule: normal
Last Changed Author: martindholmes
Last Changed Rev: 570
Last Changed Date: 2020-01-13 11:41:56 -0800 (Mon, 13 Jan 2020)

You'll see that there are two revision numbers; the last-changed is 570, but the current revision is 588. Presumably that's because the master branch last changed in 570 but the dev branch changed more recently (as you'd expect). I have a suspicion that the svn polling mechanism in Jenkins interprets this to mean that there are new changes, and does a build. But it may be some other bug.

Anyway, I'm thinking that it might be a better alternative to pull static copies of the remote repo contents during the build process. The only disadvantage of this is that you can't live-edit the remote repo's code in the context of one of the projects using it, then commit from there back into the real repo. But that really just means that you have to have solid testing components in the original repo rather than relying on the consumer projects to do testing during active development.

PostgreSQL (psql) has several xml output methods.

For example, to dump a table to an XML file:

# /usr/pgsql-9.6/bin/psql -v ON_ERROR_STOP=1 -d name_of_db
name_of_db=# COPY (SELECT table_to_xml('name_of_schema.name_of_table', true, false, '')) to '/path/to/backup.xml'

There are two problems with using XML outputs:
1) The XML is pretty basic
2) Because of the way PostgreSQL works the whole transaction is done at once so memory allocation on large tables will kill the process (or the server) and you end up with messages like this:

server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

The advice from sysadmin is don't use XML output for big things.

Find all running queries:

SELECT pid, datname, usename, query, state, now() - pg_stat_activity.query_start AS duration FROM pg_stat_activity;<br /><br />

Refresh a materialized view:

REFRESH MATERIALIZED VIEW schema_name.view_name;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA app TO user_name;
GRANT SELECT ON ALL TABLES IN SCHEMA app TO user_name;

Remember to grant select to whatever read-only users need access or they won't be able to read this view after the refresh.

I created a MySQL dump of the siberian DB and added it to a full dump of the www directory for the siberian user. The entire dump has been made available for download from the static site.

I've updated the version of adminer to 4.7.5 and added an XML dump plugin (accessed as a file type in the export form).