#Google Search Engine Robot User-agent: Googlebot Allow: /?_escaped_fragment_ Allow: /*?lang= Allow: /hashtag/*?src= Allow: /search?q=%23 Allow: /i/api/ Disallow: /search/realtime Disallow: /search/users Disallow: /search/*/grid Disallow: /*? Disallow: /*/followers Disallow: /*/following Disallow: /account/deactivated Disallow: /settings/deactivated #Yahoo! Search Engine Robot User-Agent: Slurp Allow: /?_escaped_fragment_ Allow: /*?lang= Allow: /hashtag/*?src= Allow: /search?q=%23 Allow: /i/api/ Disallow: /search/realtime Disallow: /search/users Disallow: /search/*/grid Disallow: /*? Disallow: /*/followers Disallow: /*/following Disallow: /account/deactivated Disallow: /settings/deactivated #Yandex Search Engine Robot User-agent: Yandex Allow: /?_escaped_fragment_ Allow: /*?lang= Allow: /hashtag/*?src= Allow: /search?q=%23 Allow: /i/api/ Disallow: /search/realtime Disallow: /search/users Disallow: /search/*/grid Disallow: /*? Disallow: /*/followers Disallow: /*/following Disallow: /account/deactivated Disallow: /settings/deactivated #Microsoft Search Engine Robot User-Agent: msnbot Allow: /?_escaped_fragment_ Allow: /*?lang= Allow: /hashtag/*?src= Allow: /search?q=%23 Disallow: /search/realtime Disallow: /search/users Disallow: /search/*/grid Disallow: /*? Disallow: /*/followers Disallow: /*/following Disallow: /account/deactivated Disallow: /settings/deactivated #Bing Search Engine Robot User-Agent: bingbot Allow: /?_escaped_fragment_ Allow: /*?lang= Allow: /hashtag/*?src= Allow: /search?q=%23 Disallow: /search/realtime Disallow: /search/users Disallow: /search/*/grid Disallow: /*? Disallow: /*/followers Disallow: /*/following Disallow: /account/deactivated Disallow: /settings/deactivated # Every bot that might possibly read and respect this file. User-agent: * Allow: /*?lang= Allow: /hashtag/*?src= Allow: /search?q=%23 Allow: /i/api/ Disallow: /search/realtime Disallow: /search/users Disallow: /search/*/grid Disallow: /*? Disallow: /*/followers Disallow: /*/following Disallow: /account/deactivated Disallow: /settings/deactivated Disallow: /oauth Disallow: /1/oauth Disallow: /i/streams Disallow: /i/hello # WHAT-4882 - Block indexing of links in notification emails. This applies to all bots. Disallow: /i/u Noindex: /i/u # Wait 1 second between successive requests. See ONBOARD-2698 for details. Crawl-delay: 1 # Independent of user agent. Links in the sitemap are full URLs using https:// and need to match # the protocol of the sitemap. Sitemap: https://twitter.com/sitemap.xml