User-agent: * Sitemap: https://habr.com/sitemap/index.xml User-agent: Yandex Crawl-delay: 2 Disallow: /search/ Disallow: /js/ Disallow: /css/ Disallow: /ajax/ Disallow: /login/ Disallow: /register/ Clean-param: utm_source&utm_medium&utm_term&utm_campaign / User-agent: Googlebot Disallow: /search/ Disallow: /js/ Disallow: /css/ Disallow: /ajax/ Disallow: /login/ Disallow: /register/ Disallow: /*?*utm_source= Disallow: /*?*utm_medium= Disallow: /*?*utm_term= Disallow: /*?*utm_campaign= User-agent: Slurp Crawl-delay: 8 Disallow: /search/ Disallow: /js/ Disallow: /css/ Disallow: /ajax/ Disallow: /login/ Disallow: /register/ Disallow: /*?*utm_ User-agent: * Crawl-delay: 10 Disallow: /search/ Disallow: /js/ Disallow: /css/ Disallow: /ajax/ Disallow: /login/ Disallow: /register/ Disallow: /*?*utm_