robots.txt viewer

shopee.co.th @ 2019-02-10 12:27:03


website: https://shopee.co.th
robots.txt: shopee.co.th/robots.txt

/robots.txt (816B)




User-Agent:Googlebot
User-Agent:Bingbot
Crawl-delay:0.1
Disallow: /cart/
Disallow: /checkout/
Disallow: /buyer/
Disallow: /user/
Disallow: /me/
Disallow: /order/
Disallow: /daily_discover/
Disallow: /mall/just-for-you/
Disallow: /mall/*-cat.
Disallow: /from_same_shop/
Disallow: /you_may_also_like/
Disallow: *-i.*/similar?from=flash_sale
Disallow: /find_similar_products/
Disallow: /top_products
Disallow: /search*searchPrefill

User-Agent:*
Crawl-delay:1
Disallow: /cart/
Disallow: /checkout/
Disallow: /buyer/
Disallow: /user/
Disallow: /me/
Disallow: /order/
Disallow: /daily_discover/
Disallow: /mall/just-for-you/
Disallow: /mall/*-cat.
Disallow: /from_same_shop/
Disallow: /you_may_also_like/
Disallow: *-i.*/similar
Disallow: /find_similar_products/
Disallow: /top_products
Disallow: /search*searchPrefill