On Mon, Sep 20, 2010 at 12:57 AM, John Maddock
Does anyone on this list have (or have a friend with) SEO expertise? They would know how to address this. Otherwise, Paul, maybe you could try to ask Google themselves?
The problem is that bots are currently blocked from accessing the documentation pages because they are considered too expensive. I've reduced the cost of serving the documentation pages, but apparently that's not good enough because the regression test results are still served from zipfiles. Someone needs to sort that out with whoever runs the regression testing system.
Surely the test result pages aren't accessed that much? As long as they're blocked from the bots they shouldn't take up too much CPU time even when zipped?
IMO it's unacceptible not to have the Boost documentation indexed by the web crawlers - would it be possible to allow indexing of current release *only* - and see how that goes?
Cheers, John.
A single sitemap would work for all the big search engines. A sitemap lets you specify how often something is updated -- their spiders should index it far less often if you specify a long term. You might be able to do the same thing with HTTP cache headers. Depending on your web server you might be able to configure entire subdirs to cache until the end of time. -- Cory Nelson http://int64.org