I am playing with the idea of building a search engine. I think that this idea is a bit more than I can chew but hey, what the hell, I am sure no one in their right mind has though of doing it before, not!
I had a good think about what I could do that would be a bit smaller and marginally easier to manage so I have decided to store URL’s. I know that this is still going to be a massive undertaking but I have never been one to shirk from the unsurmountable plus I get to play with lots of data something I like to do. There is no real purpose to this other than me getting to play with massive amounts of data. I also get to thrash the crap out of my NTL connection and my little box.
I have a 1Mb connection which in the UK is considered fairly fast. I am running a Athlon XP 1700 with 512Mb RAM. I have 4 hard disks one of which is an 18Gb SCSI the rest are 20Gb IDE. You can see that I am already going to run out of RAM and HD space in a fairly short space of time.