Wednesday, July 14, 2004

The bots are here...

In actually no time (just 3 days after the site is on-air), search engine spiders arrived. A couple of observations:

Yahoo!
The client string is set to Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp). The IPs used where in the following subnets: 66.196.90.*, 66.196.91.* and 66.196.101.* Yahoo! robot(s) performed 68 requests today, and always looked for robots.txt.

Google
Googlebot (identified by Googlebot/2.1 (+http://www.google.com/bot.html)) performed only 4 requests today and 2 yestarday, from the following IPs: 64.68.82.136, 64.68.82.37, 64.68.82.172. It will be interesting to see how much time will pass until I can see results when searching google...

0 Comments:

Post a Comment

<< Home