Module tokenize


Classes

StopTokenizing
TokenError

Functions

ISEOF
ISNONTERMINAL
ISTERMINAL
any
generate_tokens
group
main
maybe
printtoken
tokenize
tokenize_loop
untokenize

Variables

AMPER
AMPEREQUAL
AT
BACKQUOTE
Bracket
CIRCUMFLEX
CIRCUMFLEXEQUAL
COLON
COMMA
COMMENT
Comment
ContStr
DEDENT
DOT
DOUBLESLASH
DOUBLESLASHEQUAL
DOUBLESTAR
DOUBLESTAREQUAL
Decnumber
Double
Double3
ENDMARKER
EQEQUAL
EQUAL
ERRORTOKEN
Expfloat
Exponent
Floatnumber
Funny
GREATER
GREATEREQUAL
Hexnumber
INDENT
Ignore
Imagnumber
Intnumber
LBRACE
LEFTSHIFT
LEFTSHIFTEQUAL
LESS
LESSEQUAL
LPAR
LSQB
MINEQUAL
MINUS
NAME
NEWLINE
NL
NOTEQUAL
NT_OFFSET
NUMBER
N_TOKENS
Name
Number
OP
Octnumber
Operator
PERCENT
PERCENTEQUAL
PLUS
PLUSEQUAL
PlainToken
Pointfloat
PseudoExtras
PseudoToken
RBRACE
RIGHTSHIFT
RIGHTSHIFTEQUAL
RPAR
RSQB
SEMI
SLASH
SLASHEQUAL
STAR
STAREQUAL
STRING
Single
Single3
Special
String
TILDE
Token
Triple
VBAR
VBAREQUAL
Whitespace
__credits__
double3prog
endprogs
pseudoprog
single3prog
single_quoted
t
tabsize
tok_name
tokenprog
triple_quoted

[hide private]