# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /go/ Disallow: /ping/ Disallow: /_data/ Disallow: /_data-cached/ Disallow: /*?item=* Disallow: /*&item=* Sitemap: https://www.everysize.com/sitemap.xml # Block crawler #User-agent: AhrefsBot User-agent: SemrushBot User-agent: sistrix User-agent: Optimizer User-agent: BLEXBot User-agent: dotbot User-agent: MJ12bot User-agent: Barkrowler User-agent: seekbot User-agent: PetalBot User-agent: Screaming Frog SEO spider Disallow:/