To use the HTML5 doctype ("<!DOCTYPE html>") with an eXist app, your exist:serialize string needs to look something like this:
declare option exist:serialize "method=html5 media-type=application/xhtml+xml encoding=utf-8 indent=yes doctype-public=''";
The important bits are the method and doctype-public options. Without both of those set, the doctype will not output properly.