Following instructions yesterday from CT on the TEI list, I've implemented a system for serializing TEI XML using the new TEI mime type, with a selector which determines whether the browser can handle it or not before using it, or using text/xml instead. These are the sitemap details:

In <map:serializers>:
          <!-- MDH: added this serializer to allow for UTF-8 XML output with TEI mime type. -->
          <map:serializer mime-type="application/tei+xml" name="tei" src="org.apache.cocoon.serialization.XMLSerializer">
            <encoding>UTF-8</encoding>
          </map:serializer>

In <map:selectors>:
	            	  <map:selector name="accept-content-type" 
        		    src="org.apache.cocoon.selection.RegexpHeaderSelector">
        		    <pattern name="tei">application/tei\+xml</pattern>
        		    <header-name>accept</header-name>
        		  </map:selector>

After <map:components>:

  <map:resources>
    <map:resource name="serialize-tei">
      <map:select type="accept-content-type">
        <map:when test="tei">
          <map:serialize type="utf8tei"/>
        </map:when>
        <map:otherwise>
          <map:serialize type="utf8xml"/>
        </map:otherwise>
      </map:select>
    </map:resource>
  </map:resources>

And in actual pipelines. use <map:call resource="serialize-tei"/> instead of <map:serialize type="utf8xml"/>. For example:

            <map:match pattern="getDoc.xml">
                <map:generate src="xq/doc.xq" type="xquery"/>
                <map:transform type="saxon" src="xsl/highlight_matches.xsl">
                    <map:parameter name="browserURI" value="{request:requestURI}?{request:queryString}"/>
                    <map:parameter name="queryString" value="{request:queryString}"/>
                </map:transform>
                <!--<map:transform type="saxon" src="xsl/add_xml_stylesheet.xsl" />-->
                <map:transform type="xinclude"/>
                <map:transform type="session"/>
                <map:transform type="encodeURL"/>
              <map:call resource="serialize-tei"/>
            </map:match>

For Firefox, you can set the browser to handle the TEI mime type in preference to the text/xml alternative by changing network.http.accept.default in about:config. This is my setting:

text/html;q=0.7,application/xhtml+xml;q=0.7,application/xml;q=0.7,text/xml;q=0.8,application/tei+xml;q=0.2,*/*;q=0.1

"q" settings are between 0 and 1, with higher priority for higher values, so here application/tei+xml is higher priority than text/xml.

To decide what happens to the mime type when the browser encounters it, you have to let the browser encounter it (unless you install the MIME Edit extension, which gives you actual control over mime type handling).

Other browsers are more problematic. This system works on the basis that the browser provides a prioritized list of acceptable mime types to the server (Cocoon), which can then serve application/tei+xml if the browser handles it. However, Chrome does not allow you to configure acceptable mime types, so it doesn't seem possible to make it announce to Cocoon that application/tei+xml is acceptable; therefore Cocoon will never deliver application/tei+xml to Chrome. If it were possible to deliver the correct mime type to Chrome, it would just hand it off to the OS or desktop (Gnome etc.) to deal with. Opera does allow you to configure what it will do with mime types, so you can set (for instance) application/tei+xml to open with oXygen instead of being displayed in the browser, but this does not appear to affect the list of mime types Opera sends to the server, so Cocoon is still delivering text/xml (as far as I know).

IE8 appears to do all file handling based on file extensions, which is not really helpful; there are hacks you can do in the registry, but it's not clear to me that they would succeed in achieving anything unless the file were delivered with a specific unique extension. IE9 beta is no different in this respect.