I have seen plenty of robots spider my site but by far the most polite as far as time goes has got to be the miragorobot. It waited a full minute between each page request which is almost unheard of.
Parent, Painter, Programmer
I have seen plenty of robots spider my site but by far the most polite as far as time goes has got to be the miragorobot. It waited a full minute between each page request which is almost unheard of.