aboutsummaryrefslogtreecommitdiffstats
path: root/public/robots.txt
Commit message (Expand)AuthorAgeLines
* Update references to point to the mySociety repo.Louise Crow2012-08-29-1/+1
* Fix robots.txt to handle URLs with localesDavid Cabo2012-05-08-11/+12
* Discourage spiders from crawling old, wrongly generated URLs as per issue #311.Seb Bacon2012-01-11-0/+6
* Add /body/*/view_email$ to robots.txtRobin Houston2011-08-31-1/+5
* Add /signing to the list of disallowed paths in robots.txt, because Googlebot...Robin Houston2011-07-28-0/+2
* Change all URLs for users to be under /profile so neaterFrancis Irving2010-07-20-1/+1
* Remove Bing crawl delayfrancis2009-09-17-2/+0
* Slow it for now while we speed up page serving.francis2009-09-02-0/+5
* Block /feed/ alsofrancis2009-09-02-0/+6
* Block a few more URLs that don't need to go in search engines.francis2009-09-02-0/+4
* Removed some bits there is no need to be in search engines.francis2009-09-02-0/+2
* Move similar request URLs, so can block in robots.txt easily, and block them ...francis2008-09-15-1/+5
* Default Rails gumph.francis2007-08-01-0/+1