robots.txt details

saved: size: 423 kb md5 checksum: 0c831251aea143f9111c71f3a3a695e5

websites using this robots.txt

content


User-agent: *
Disallow: /my/
Crawl-delay: 3
Sitemap: https://spb-dvigenie.ru/sitemap.70865.xml.gz

User-Agent: Yandex
Allow: /
Disallow: /my/
Crawl-delay: 3
Host: https://spb-dvigenie.ru
Sitemap: https://spb-dvigenie.ru/sitemap.70865.xml.gz

User-agent: msnbot
Crawl-delay: 10

User-agent: MJ12bot
Crawl-delay: 20

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /