robots.txt details

19b094bc456e65bc724ece3f0b67f266

saved: | size: 998kb | checksum: 19b094bc456e65bc724ece3f0b67f266

websites using this robots.txt

content


User-agent: *

##### Transactional and Content
Disallow: /on/demandware.store/*

Disallow: *Login-*
Disallow: *Register-*
Disallow: *Wishlist-*
Disallow: *Cart-*
Disallow: *Checkout-*
Disallow: *Account-*
Disallow: *Order-*

Disallow: /*null*

##### Search and Query Params
Allow: /*?dwvar
Allow: /*utm_medium=merchant
Disallow: /*pmin*
Disallow: /*pmax*
Disallow: /*srule=*
Disallow: /*?q=*
Disallow: /*prefv*=*
Disallow: /*prefn*=*

##### Slow crawling rate for some agents
User-agent: msnbot
Crawl-delay: 30

User-agent: bingbot
Crawl-delay: 30

##### Disallow aggressive agents
User-agent: MJ12bot
Disallow: /

User-Agent: trendictionbot
Disallow: /

user-agent: MegaIndex.ru
Disallow: /

User-agent: dotbot
Disallow: /

user-agent: 360Spider
Disallow: /

user-agent: HaosouSpider
Disallow: /

user-agent: YandexBot
Disallow: /

user-agent: Baiduspider
Disallow: /


##### Sitemaps
Sitemap: https://www.multipharma.be/sitemap_index.xml