diff options
| author | Matthew Somerville <matthew@mysociety.org> | 2010-12-21 17:25:23 +0000 |
|---|---|---|
| committer | Matthew Somerville <matthew@mysociety.org> | 2010-12-21 17:25:23 +0000 |
| commit | c40e7cdb33f06bdb7465089f100a7a44bcbfd730 (patch) | |
| tree | 7ff8da4e9867ace40aff9f7496214f439f3c2b9a | |
| parent | be02fdfda244dd18b958dd0759369d842bca00ab (diff) | |
Dear bingbot, please stop hammering our server. Thanks.
| -rw-r--r-- | web/robots.txt | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/web/robots.txt b/web/robots.txt index e69de29bb..53675186c 100644 --- a/web/robots.txt +++ b/web/robots.txt @@ -0,0 +1,5 @@ +User-agent: msnbot +Crawl-delay: 3 + +User-agent: bingbot +Crawl-delay: 3 |