# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead. # Please see https://developer.wordpress.com/docs/firehose/ for more details. Sitemap: https://techcrunch.com/news-sitemap.xml User-agent: Swiftbot Disallow: User-agent: Twitterbot Disallow: User-agent: * Disallow: /search/ Disallow: */lightbox Sitemap: https://techcrunch.com/video-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Sitemap archive Sitemap: https://techcrunch.com/sitemap.xml Disallow: /wp-login.php Disallow: /activate/ # har har Disallow: /cgi-bin/ # MT refugees Disallow: /mshots/v1/ Disallow: /next/ Disallow: /public.api/ User-agent: IRLbot Crawl-delay: 3600 # This file was generated on Thu, 19 Oct 2017 06:50:05 +0000