User-agent: * Disallow: /my/ Crawl-delay: 3 Sitemap: http://afs-tuning.ru/sitemap.1319973.xml.gz User-Agent: Yandex Allow: / Disallow: /my/ Crawl-delay: 3 Host: afs-tuning.ru Sitemap: http://afs-tuning.ru/sitemap.1319973.xml.gz User-agent: msnbot Crawl-delay: 10 User-agent: MJ12bot Crawl-delay: 20 User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: /