robots.txt details

saved: size: 385 kb md5 checksum: a2edd8dbf3af910845739233d069049d

websites using this robots.txt

content


Sitemap: https://www.getreading.co.uk/map_news.xml
Sitemap: https://www.getreading.co.uk/sitemaps/sitemap_index.xml
User-agent: *
Disallow: /topics/*
Disallow: /topics
Disallow: /*token=*
Disallow: /search/
Disallow: /comm-part-test/
Disallow: /*service=ajax
Disallow: /*.ece
Disallow: /5293/
Crawl-delay: 10.0

User-agent: semetrical
Disallow: /

User-agent: grapeshot
Crawl-delay: 0