Finally found a fix to make the sidebars long enough in IE6. Basically, you set it like this:

var heightForSidebars = document.body.scrollHeight;
        if (document.body.offsetHeight > heightForSidebars){
            heightForSidebars = document.body.offsetHeight;
        }

This sets the height to the height of the document, or the height of the window, whichever is larger; when it's a long page, the former (scrollHeight) is larger, and when there's little content, the latter may be longer.

On the actual ScanCan site, the problems are slightly more complicated, because there are lots of interactions during the JavaScript that lays out the page. If the content of the scrollbars changes during the layout, then we need to check their height to see if either of them is now taller than the heightForSidebars we've already calculated, and take the largest.

I also had to add a call to the CheckCompatibility() function in the ajax.js file, so that when the AJAX code writes material to the page, the page is laid out again, since its length will often change.

The results are good enough for me to feel confident in suppressing the warning for IE6 users. The site basically seems to function correctly on IE6 with this compatibility code.