diff options
| author | Louise Crow <louise.crow@gmail.com> | 2013-12-09 12:16:30 +0000 |
|---|---|---|
| committer | Louise Crow <louise.crow@gmail.com> | 2013-12-09 12:16:30 +0000 |
| commit | f2a0c9485dd2655c11e9717583c7d3c7b3a49d7a (patch) | |
| tree | 0949c25085d4c8a093fda051e1497570b968c6c7 | |
| parent | 96d72a6ea4ba3a46c1a0bb7eb9be84013660a558 (diff) | |
| parent | 3e5696d71df5a2726f58ea0c446b68055924c2d7 (diff) | |
Merge branch 'feature/1233-allow-crawling-of-attachments' into rails-3-develop
| -rw-r--r-- | public/robots.txt | 6 |
1 files changed, 5 insertions, 1 deletions
diff --git a/public/robots.txt b/public/robots.txt index 279573d31..969e10826 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -8,7 +8,10 @@ # Crawl-delay: 1 # http://www.bing.com/community/blogs/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx -# This file uses the non-standard extension characters * and $, which are supported by Google and Yahoo! +# This file uses the non-standard extension characters * and $, which +# are supported by Google and Yahoo! and the 'Allow' directive, which +# is supported by Google. + # http://code.google.com/web/controlcrawlindex/docs/robots_txt.html # http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-02.html @@ -23,6 +26,7 @@ Disallow: */user/contact/ Disallow: */feed/ Disallow: */profile/ Disallow: */signin +Allow: */request/*/response/*/attach/* Disallow: */request/*/response/ Disallow: */body/*/view_email$ |