Easy to use extension of Net::HTTP to let you obey robots.txt while crawling/scrapping/mining
>= 0
Abhishek Parolkar