Arachnid is a web crawler that relies on Bloom Filters to efficiently store visited urls and Typhoeus to avoid the overhead of Mechanize when crawling every page on a domain.
Required Ruby Version
None
Authors
dchuk
Versions
- 0.4.1 January 17, 2014 (5 KB)
- 0.4.0 January 16, 2014 (4.5 KB)
- 0.3.2 January 16, 2014 (5 KB)
- 0.3.1 January 16, 2014 (5 KB)
- 0.3.0 January 16, 2014 (4.5 KB)