diff options
| author | David Cabo <david@calibea.com> | 2012-04-13 02:13:38 +0200 |
|---|---|---|
| committer | Seb Bacon <seb.bacon@gmail.com> | 2012-05-08 13:39:33 +0100 |
| commit | 27a8cb109104d0cfe2677930f78870916aa1147b (patch) | |
| tree | de1d894a52b772c8b84ad6d5e8ce5a068c475824 | |
| parent | cb9974038b654457578dc3dadeb33d5e6f01f759 (diff) | |
Fix robots.txt to handle URLs with locales
| -rw-r--r-- | public/robots.txt | 23 |
1 files changed, 12 insertions, 11 deletions
diff --git a/public/robots.txt b/public/robots.txt index 6a8628c93..a187d0775 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -13,17 +13,18 @@ # http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-02.html User-agent: * -Disallow: /annotate/ -Disallow: /new/ -Disallow: /search/ -Disallow: /similar/ -Disallow: /track/ -Disallow: /upload/ -Disallow: /user/contact/ -Disallow: /feed/ -Disallow: /profile/ -Disallow: /signin -Disallow: /body/*/view_email$ +Disallow: */annotate/ +Disallow: */new/ +Disallow: */search/ +Disallow: */similar/ +Disallow: */track/ +Disallow: */upload/ +Disallow: */user/contact/ +Disallow: */feed/ +Disallow: */profile/ +Disallow: */signin +Disallow: */request/*/response/ +Disallow: */body/*/view_email$ # The following adding Jan 2012 to stop robots crawling pages # generated in error (see |