# Pantheon's documentation on robots.txt: https://pantheon.io/docs/bots-and-indexing/ User-agent: * Disallow: / User-agent: RavenCrawler User-agent: rogerbot User-agent: dotbot User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: PowerMapper User-agent: Swiftbot Allow: /