Module tokenize
Classes
Functions
ISEOF
ISNONTERMINAL
ISTERMINAL
generate_tokens
main
tokenize
untokenize
Variables
AMPER
AMPEREQUAL
AT
BACKQUOTE
CIRCUMFLEX
CIRCUMFLEXEQUAL
COLON
COMMA
COMMENT
DEDENT
DOT
DOUBLESLASH
DOUBLESLASHEQUAL
DOUBLESTAR
DOUBLESTAREQUAL
ENDMARKER
EQEQUAL
EQUAL
ERRORTOKEN
GREATER
GREATEREQUAL
INDENT
LBRACE
LEFTSHIFT
LEFTSHIFTEQUAL
LESS
LESSEQUAL
LPAR
LSQB
MINEQUAL
MINUS
NAME
NEWLINE
NL
NOTEQUAL
NT_OFFSET
NUMBER
N_TOKENS
OP
PERCENT
PERCENTEQUAL
PLUS
PLUSEQUAL
RBRACE
RIGHTSHIFT
RIGHTSHIFTEQUAL
RPAR
RSQB
SEMI
SLASH
SLASHEQUAL
STAR
STAREQUAL
STRING
TILDE
VBAR
VBAREQUAL
tok_name
[hide private]