User-agent: * Allow: / Crawl-delay: 3 User-agent: bingbot Crawl-delay: 30 User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: JoobleBot Disallow: / User-agent: Fasterfox Disallow: / User-Agent: Yandex Disallow: / User-Agent: YandexBot Disallow: / User-agent: Exabot Disallow: / User-agent: proximic Disallow: / User-Agent: YoudaoBot Disallow: / User-Agent: AhrefsBot disallow: /