I just got back from N Ireland and sat down today and drew up some plans. I have Oracle Postgres and MySQL to choose from as a backend to this system.
Oracle
Those who have worked with Oracle will know that on a system like mine you would be lucky to get it running due to memory requirements. I have it installed and working but I had to raise the shred memory to over 300Mb to get it to work properly. This does not leave much room for anything else on my box. There is also the license issue. I may want to put the database online some time which means lots of cash.
Postgres
This is a fantastic database. It has a reasonable memory footprint for small apps and can scale very well. I have never thrashed it but according to reports 7.3 is very fast compared to 7.2.
MySQL
I am afraid that this database although a very good product has several features missing that I cannot live without.
I have decided to use Postgres. I have also decided that the robots will be written in Perl. The reasons for using Perl are quite simple, I know it and its versatile, you can pretty much do anything you want in it. For those die hards who may claim that it wont be fast enough remember I am not limited by the robots but several other factors like bandwidth etc. The robots will utilise some pre-packaged modules designed just for the task at hand. I will post the code at some point.
–These are the tables that I am going to be using
create table home_page
(
url_id int4 DEFAULT NEXTVAL(‘url_id_pk’),
url varchar(2000) unique,
state int4 default 0,
date_time timestamp default now(),
last_modified_time timestamp,
PRIMARY KEY(url_id, url)
);
create table links_found
(
parent_url_id int4,
found_url varchar(2000),
PRIMARY KEY(parent_url_id, found_url)
);