diff options
| author | David Cabo <david@calibea.com> | 2011-10-13 00:29:51 +0200 |
|---|---|---|
| committer | David Cabo <david@calibea.com> | 2011-10-13 00:29:51 +0200 |
| commit | c8983b923e4dc7db9ba22156daaddd94d2b5ed4d (patch) | |
| tree | 7741c3655fe5e3cbc90dd20a4626ac7acc1bf6b0 /public/robots.txt | |
| parent | a29b3aaf0ae77af49d38813b62dddcb6889c1ebe (diff) | |
| parent | e13127a8ebc8bf8379d92f778af5a2bb6931d80c (diff) | |
Merge branch 'release/0.4'0.4
Diffstat (limited to 'public/robots.txt')
| -rw-r--r-- | public/robots.txt | 6 |
1 files changed, 5 insertions, 1 deletions
diff --git a/public/robots.txt b/public/robots.txt index 80dfd8ef7..029ae0dbf 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -8,6 +8,10 @@ # Crawl-delay: 1 # http://www.bing.com/community/blogs/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx +# This file uses the non-standard extension characters * and $, which are supported by Google and Yahoo! +# http://code.google.com/web/controlcrawlindex/docs/robots_txt.html +# http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-02.html + User-agent: * Disallow: /annotate/ Disallow: /new/ @@ -19,4 +23,4 @@ Disallow: /user/contact/ Disallow: /feed/ Disallow: /profile/ Disallow: /signin - +Disallow: /body/*/view_email$ |