Figured this out after seeing a sample XEP config for Oxygen on the Oxygen forum (thanks Adrian).

In the process of updating Ubuntu, a bunch of apps got removed because of repo problems. I reinstalled all the ones I noticed, but it looks like the "xep" script file that runs the app was deleted. That would have presented Oxygen with a problem, presumably, and it seems to have responded by setting the XEP config to this:

/media/Data/mholmes/apps/RenderX/XEP/lib/xep.jar -fo ${fo} -${method} ${out}

In other words, it was pointing directly at the jar file, not at the now-missing script. I've restored the xep startup file from a backup, and pointed Oxygen at it, and now everything works.

The XEP error message didn't help figure this out.