robots.txt details

8ce9c9730f926bbbae922c741007438b

saved: | size: 744kb | checksum: 8ce9c9730f926bbbae922c741007438b

websites using this robots.txt

content


User-agent: *
Disallow: /sorry.aspx
Disallow: /error.aspx
Disallow: /main/banner.asp
Disallow: /advertisers/linkclick.asp
Disallow: /products/productretailer.asp
Disallow: /community/classifieds/resend_email.asp
Disallow: /Library/LogAction.aspx
Disallow: /Library/MailshotTracking.aspx
Disallow: /Library/PressReleaseClick.aspx
Disallow: /nl/redirect.aspx
Disallow: /nl/link.aspx
Disallow: /redirect/articleid.aspx
Disallow: /redirect/indexs.aspx
Disallow: /redirect/photo.aspx

User-agent: AhrefsBot
Disallow: /forum/
Crawl-delay: 5

User-agent: MJ12bot
Disallow: /forum/
Crawl-delay: 5

User-agent: SemrushBot
Disallow: /forum/
Crawl-delay: 5

User-agent: DotBot
Disallow: /

User-agent: Linguee
Disallow: /