robots.txt viewer

www.zeit.de @ 2019-01-10 09:22:02


website: http://www.zeit.de
robots.txt: www.zeit.de/robots.txt

/robots.txt (387B)

User-agent: Googlebot-News
Disallow: /angebote/

User-agent: *
Disallow: /zeit/
Disallow: /templates/
Noindex: /templates/
Disallow: /hp_channels/
Disallow: /send/
Disallow: /suche/
Noindex: /suche/
Disallow: */comment-thread?
Noindex: */comment-thread?

User-agent: Baiduspider
Disallow: /

User-agent: GrapeshotCrawler
crawl-delay: 3

Sitemap: https://www.zeit.de/gsitemaps/index.xml