Finished the first Robot

I have just finished the first Robot and have seeded the database with two links and and started it running. I am going to start writing another robot that checks the headers of the webpages. I am doing this so that I can spot errors and mark them not to be retrieved. I also need to compile a list of document types that I do not want to download.
120K links_found
120K home_page

Leave a Reply

Your email address will not be published. Required fields are marked *