# www.robotstxt.org/ # XML sitemap # Sitemap: https://www.domain.com/sitemap_index.xml # Allow crawling of all content User-agent: * Disallow: # WordPress specific rules Disallow: /wp-content/languages/ Disallow: /wp-content/cache/ Disallow: /wp-content/w3tc-config/ Disallow: /wp-content/upgrade/ # Disallow annoying bots User-agent: Linguee bot Disallow:/