robots.txt details

saved: size: 1182 kb md5 checksum: 9d621f1920c932a6ca87abb17e8aef01

websites using this robots.txt

content


User-agent: *
Disallow: /gateway/
Disallow: /thorntree/*
Disallow: /members/*
Disallow: /assets/
Disallow: /*?create_review=true*

Disallow: */v/vid/*?tag
Disallow: /*/articles/category/*
Disallow: *?type
Disallow: *?sort
Disallow: *?subtype
Disallow: *?neighborhoods
Disallow: */pa-act/
Disallow: */pa-tou/
Disallow: */poi-sho/
Disallow: */poi-ent/
Disallow: */poi-sle/
Disallow: */poi-dri/
Disallow: */poi-eat/

Disallow: *searchResult?q
Disallow: *search?q
Disallow: *?utf8
Disallow: *?search
Disallow: *?&search
Disallow: *&search
Disallow: *?sort_dir
Disallow: *?sort_order
Disallow: *?filters
Disallow: *?ab_tours_cta
Disallow: *?authenticity_token
Disallow: *?ab_viator_activity_cta
Disallow: *?lpaffil
Disallow: *?category

Sitemap: https://www.lonelyplanet.com/articles/sitemap.xml
Sitemap: https://www.lonelyplanet.com/video/smap.xml
Sitemap: https://www.lonelyplanet.com/articles/google-news
Sitemap: https://www.lonelyplanet.com/stories/sitemap-index.xml

User-agent: Sphere (www.sphere.com) scout*at*sphere*dot*com
Crawl-delay: 0.5

User-agent: BLEXBot
Crawl-delay: 10

User-agent: Blekkobot
Crawl-delay: 10

User-agent: 008
Disallow: /

User-agent: voltron
Disallow: /