Since September 3rd, a bot at Yahoo (68.142.195.81) has been repeatedly, as in 45,266 times and still counting, polling my web site. I’ve finally added the magic line in .htaccess, but it’s annoying that there’s no way to contact a human being at yahoo to kick this thing in the pants.
Their bot is requesting pages from my site every five seconds, ignoring the 10-second interval specified in my robots.txt file, and oscillating among three pages. Repeat this a gazillion times:
68.142.195.81 – – [05/Sep/2005:13:09:15 -0700] “GET /a/2005/08/raw_2005_wow.shtml HTTP/1.0” 403 – “-” “YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)”
68.142.195.81 – – [05/Sep/2005:13:09:20 -0700] “GET /a/2005/09/post_your_first_1.shtml HTTP/1.0” 403 – “-” “YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)”
68.142.195.81 – – [05/Sep/2005:13:09:25 -0700] “GET /a/2005/08/ride_around_was_1.shtml HTTP/1.0” 403 – “-” “YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)”
Testing, indeed.
Irony alert! – after I posted this, the search bot soon changed its behavior to indexing this very blog entry over and over and over. At least it’s doing it once every 11 seconds.