linux-users archive

OT: Badly behaved spidering bots


New Message Reply Date view Thread view Subject view Author view

From: Joe Landman (email_suppressed_at_lugwash.org)
Date: Sat 15-Oct-2005 01:49:25 PM EDT


Hi folks:

   Thought you might like to have a look out for a robot from Fast
Search & Transfer at IP address 69.25.71.12 (though it looks like they
have the 69.25.71.0 - 69.25.71.255 range).

   These folks win the award for the most insidious robot. Not only do
they not respect robots.txt, they don't provide a bot name, and for that
matter, they seem to get stuck in things like calendars. Previously it
was a toss up between MSN and some other spidering from China that were
the worst offenders, but these folks take the cake by at least an order
of magnitude. They seem to like consuming our bandwidth, so if the
rule we just added doesn't deflect them, then we will block them at the
firewalls.

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: [e-mail suppressed]
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615
--
***  Sent from [e-mail suppressed]  ***  http://www.lugwash.org
to unsubscribe: `echo "unsubscribe" | mail [e-mail suppressed]`

New Message Reply Date view Thread view Subject view Author view

This archive was generated by hypermail 2.1.5 : Tue 01-Nov-2005 01:00:01 AM EST