User-agent: * Disallow: # too many repeated hits, too quick User-agent: litefinder Disallow: / # Yahoo. too many repeated hits, too quick User-agent: Slurp Disallow: / # too many repeated hits, too quick User-agent: Baidu Disallow: /