Not sure if anyone else has noticed recently but Slurp has been doing a lot of of Spidering lately. I actually took extreme measures today and added a crawl delay to the site.
Parent, Painter, Programmer
Not sure if anyone else has noticed recently but Slurp has been doing a lot of of Spidering lately. I actually took extreme measures today and added a crawl delay to the site.