User-agent: * Disallow: /products/ Disallow: /?s=* Disallow: /*?post_type* Disallow: /search/ <-- Protection of frog team --> User-agent: pesticides Disallow: /frogs/ User-agent: toads Disallow: /frogs/ User-agent: common salt Disallow: /frogs/ User-agent: the french Disallow: /frogs/legs/ <-- Protection of frog teams sanity --> User-agent: recruitment agencies Disallow: /frogs/phones/ <-- Screaming Frog - Search Engine Marketing --> If you're looking at our robots.txt then you might well be interested in our current SEO vacancies :-) www.screamingfrog.co.uk/careers/ ........?$$$M$$$$$$$$$$$$$$$$$$......... .. ..$$$MM$$M$M$$$$$$$$$N$$$$$$$. . .. ....$$$$$$7MMM.M.......MMMMM$$$$$$$..... ...$$$$$$. MMMM ...MMMMMMMMMM$$$$=.. ..$$$$$.. MMM MMMMMMMMMMM.$$$$$I.. .$$$$$... MMM$MMMMMMMMMMMMM. .$$$$$.. $$$$$.... .$$MMMOMMMMMMMMMM .. ..$$$$$. $$$$..... $$$$8MMMMMMMMMMMM........$$$$. $$$$.. ..$$$$$$.MMMMMMMMMMM....... $$$$, $$MM .M .$$$$= IMMMMMMMM$$M ...M..MM$$$7 $$$$MM7M?MMM$ MMMMMMMM$$$MM...M.M~$$$$7 $MMMMMMMIMMMM MMMMMMMZ$$$.MMMMMMMMMMM$$ $$$$..MMMMMMMMMMMMMMMM$$$$. MMMMMMM$$$$$ $$$$. .MMMMMMMMMMMMMM.$$$$. . ..$$$$7 $$$$....MMMMMMMMMMMM .$$$$.. . . $$$$7 $$$$.... MMMMMMMMM .$$$$$. . . . .$$$$$ $$$$......MMMMMMMMM$$$$$$$$$=====7$$$$$7 $$$$$ ..MODMMMMMMMM.$$$$$$$$$$$$$$$$$. =$$$$........$$$MMMMMMM$$$$$$$$$$$$$$$. .$$$$$7............MMMMMM . . .. . ..$$$$$$........ .. ..MMM . . ...$$$$$$$,........ ..MMMM . .. .. . .....$$$$$$$$$$$$$$$MMMM$$$$$MM. . . . .......$$$$$$$$$$$$MMMMM$$M$MM$. . ... .... .+$$$$$$DMMM$M8MMMMM$$......... .... . .. . .MMMMMMMMMMMMMMM. . .. .... . . ...+. . ... ....