diff options
| author | francis <francis> | 2009-09-02 14:42:46 +0000 |
|---|---|---|
| committer | francis <francis> | 2009-09-02 14:42:46 +0000 |
| commit | 496972afdc9787f67d754029fe15af14011829b3 (patch) | |
| tree | a6cbff873c474d23203afb57678f96f29797aabf | |
| parent | eed6add13064d06e21d3096f0f78abac0c2eaebf (diff) | |
Block /feed/ also
| -rw-r--r-- | public/robots.txt | 6 |
1 files changed, 6 insertions, 0 deletions
diff --git a/public/robots.txt b/public/robots.txt index 8e47db0f7..dbbe680e0 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,5 +1,9 @@ # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file +# Mainly to reduce server load from bots, we block pages which are actions, and +# searches. We also block /feed/, as RSS readers (rightly, I think) don't seem +# to check robots.txt. + User-agent: * Disallow: /annotate/ Disallow: /new/ @@ -7,5 +11,7 @@ Disallow: /search/ Disallow: /similar/ Disallow: /track/ Disallow: /upload/ +Disallow: /user/contact/ +Disallow: /feed/ Disallow: /signin |