blob: 80dfd8ef70e86e941c8d3fa0f5fd6a80e5e1fe60 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
# Mainly to reduce server load from bots, we block pages which are actions, and
# searches. We also block /feed/, as RSS readers (rightly, I think) don't seem
# to check robots.txt.
# Note: Can delay Bing's crawler with:
# Crawl-delay: 1
# http://www.bing.com/community/blogs/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
User-agent: *
Disallow: /annotate/
Disallow: /new/
Disallow: /search/
Disallow: /similar/
Disallow: /track/
Disallow: /upload/
Disallow: /user/contact/
Disallow: /feed/
Disallow: /profile/
Disallow: /signin
|
