Turns out that the culprit was saxon.jar. When installing xep a saxon.jar gets inserted in to the WEB-INF/lib directory. The file also exists in the {xep-home}/lib directory, and (we suspect) this causes some kind of collision. The result is that when large files go in to eXist we get a generic "network error" with nothing in the logs to suggest what is causing the problem.

The question remains, why does eXist throw an error when there are 2 saxon.jar files ONLY when you try to add a large document?

So, don't put saxon.jar in to {cocoon-home}/WEB-INF/lib.
OR
Put it in, but remove it from the xep/lib directory (this has not been tested).

Everything seems to be sorted other than the {user-dir}/webapps business. I have cocoon looking in the right place, but the hack that Peter van H wrote:
<map:pipeline>
<map:match pattern="cocoon/**">
<map:generate src="cocoon://{1}"/>
<map:serialize type="xml"/>
</map:match>
</map:pipeline>
throws an error. This needs further testing.