This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
|
|
|
mtime(self)
Returns the time the robots.txt file was last fetched. |
|
|
|
modified(self)
Sets the time the robots.txt file was last fetched to the current
time. |
|
|
|
set_url(self,
url)
Sets the URL referring to a robots.txt file. |
|
|
|
read(self)
Reads the robots.txt URL and feeds it to the parser. |
|
|
|
|
|
parse(self,
lines)
parse the input lines from a robots.txt file. |
|
|
|
can_fetch(self,
useragent,
url)
using the parsed robots.txt decide if useragent can fetch url |
|
|
|
|