Module robotparser
[hide private]
[frames] | no frames]

Module robotparser

robotparser.py

Copyright (C) 2000 Bastian Kleineidam

You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2

The robots.txt Exclusion Protocol is implemented as specified in http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html

Classes [hide private]
RobotFileParser
This class provides a set of methods to read, parse and answer questions about a single robots.txt file.
RuleLine
A rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.
Entry
An entry has one or more user-agents and zero or more rulelines
URLopener
Functions [hide private]
 
_debug(msg)
 
_check(a, b)
 
_test()
Variables [hide private]
  debug = 0

Imports: urlparse, urllib