diff options
| author | francis <francis> | 2009-09-02 15:44:12 +0000 |
|---|---|---|
| committer | francis <francis> | 2009-09-02 15:44:12 +0000 |
| commit | b6418bfd23af2263d5b3e5e13638d47171a6581d (patch) | |
| tree | 1719ca969013676273628be02c2b8c3a2d7480c6 | |
| parent | ea04e51df98c18846b314e1a414b0ba9fb1cd8ec (diff) | |
Slow it for now while we speed up page serving.
| -rw-r--r-- | public/robots.txt | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/public/robots.txt b/public/robots.txt index dbbe680e0..d68f5f874 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -4,6 +4,10 @@ # searches. We also block /feed/, as RSS readers (rightly, I think) don't seem # to check robots.txt. +# Note: Can delay Bing's crawler with: +# Crawl-delay: 1 +# http://www.bing.com/community/blogs/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx + User-agent: * Disallow: /annotate/ Disallow: /new/ @@ -14,4 +18,5 @@ Disallow: /upload/ Disallow: /user/contact/ Disallow: /feed/ Disallow: /signin +Crawl-delay: 1 |