Re: [Boost-users] Could searching and indexing Boost docs mightwork better?

-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users- bounces@lists.boost.org] On Behalf Of Cory Nelson Sent: Monday, September 20, 2010 9:53 AM To: boost-users@lists.boost.org Subject: Re: [Boost-users] Could searching and indexing Boost docs mightwork better?

On Mon, Sep 20, 2010 at 12:57 AM, John Maddock wrote:

...

...

Does anyone on this list have (or have a friend with) SEO expertise? They would know how to address this. Otherwise, Paul, maybe you could try to ask Google themselves?

...

The problem is that bots are currently blocked from accessing the documentation pages because they are considered too expensive. I've reduced the cost of serving the documentation pages, but apparently that's not good enough because the regression test results are still served from zipfiles. Someone needs to sort that out with whoever runs the regression testing system.

Surely the test result pages aren't accessed that much? As long as they're blocked from the bots they shouldn't take up too much CPU time even when zipped?

IMO it's unacceptible not to have the Boost documentation indexed by the web crawlers - would it be possible to allow indexing of current release *only* - and see how that goes?

I agree that this should be the thing we try to achieve immediately.

A single sitemap would work for all the big search engines. A sitemap lets you specify how often something is updated -- their spiders should index it far less often if you specify a long term.

You might be able to do the same thing with HTTP cache headers. Depending on your web server you might be able to configure entire subdirs to cache until the end of time.

Long term, site maps looks the *right* way to do this - but it would best be done by someone who knows what they are doing (definitely NOT me!!). http://en.wikipedia.org/wiki/Site_map http://www.sitemaps.org/ Shall I start a new thread on boost-users asking for help on generating a site map? Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com