robots.txt viewer

lloydslist.maritimeintelligence.informa.com @ 2019-01-11 17:16:09


website: https://lloydslist.maritimeintelligence.informa.com
robots.txt: lloydslist.maritimeintelligence.informa.com/robots.txt

/robots.txt (937B)

User-agent: MJ12bot
Disallow: /
User-agent: *
Disallow:/rss/
User-agent: *
Disallow:/data-tools/
User-agent: * 
Crawl-delay: 1
User-agent:*
Sitemap: https://pink.pharmaintelligence.informa.com/sitemap.xml
Sitemap: https://scrip.pharmaintelligence.informa.com/sitemap.xml
Sitemap: https://rose.pharmaintelligence.informa.com/sitemap.xml
Sitemap: https://medtech.pharmaintelligence.informa.com/sitemap.xml
Sitemap: https://invivo.pharmaintelligence.informa.com/sitemap.xml
Sitemap: https://lloydslist.maritimeintelligence.informa.com/sitemap.xml
Sitemap: https://insuranceday.maritimeintelligence.informa.com/sitemap.xml
Sitemap: https://iegvu.agribusinessintelligence.informa.com/sitemap.xml
Sitemap: https://agrow.agribusinessintelligence.informa.com/sitemap.xml
Sitemap: https://iegpolicy.agribusinessintelligence.informa.com/sitemap.xml
Sitemap: https://animalpharm.agribusinessintelligence.informa.com/sitemap.xml