A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.
Required Ruby Version
None
Authors
James Edward Gray II, Jeremy Friesen, Edgar Gonzalez
A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.
None
James Edward Gray II, Jeremy Friesen, Edgar Gonzalez